Python append variable to ever occurrence of a substring

Using the function from Stachoverflow,

‘s’ is the original string, ‘sub’ the substring to be replaced, ‘repl’ the string to replace it with and ‘nth’ the nth occurence of the substring in the string ‘s’.

def nth_repl(s, sub, repl, nth):
find = s.find(sub)
i=find != -1
while find != -1 and i != nth:
find = s.find(sub, find+1)
i+=1
if i == nth:
return s[:find]+repl+s[find+len(sub):]
return s

Placing this function in a loop which will iterate through every occurrence of the substring in the string and append the index of each nth occurrence with that index.

for i in range(0,num+1):
newstr = nth_repl(str, "text", "text%d"%(i-1), i)
str = newstr

And then,

print str

Hardening SSH

SSH created key-pair

More information on this topic at securing-your-server.
On your local machine, create a 4096-bit RSA key-pair,

$ ssh-keygen -b 4096

If you get a message to say that this already exists or to over write a file, then do not proceed, unless you want to create a new key-pair. If one exists, then it is probably being used to ssh into somewhere else. Don’t want to mess that up!!!
To check if it already exists,

$ cat ~/.ssh/id_rsa

Copy key to remote server

Next copy this key to the remote server,

$ ssh-copy-id @

Now login to the remote server. You should not be asked for the password, since the key-pair now exists.

We do not want root logins. Open,

/etc/ssh/sshd_config and navigate to ‘#Authentication:’ and change to,

PermitLogin no

and also change to this, further down,

# Change to no to disable tunnelled clear test passwords
PasswordAuthentication no

Raspberry Pi Linux – Managing User Accounts

This information is available on the Rspberry Pi site here. I am just recapping for myself.

Change a user’s password

$ sudo passwd

Remove a user’s password

$ sudo passwd -d

Create a new user

$ sudo adduser

Remove a user,

$ sudo userdel -r

‘pi’ is a ‘sudoer’, which means this user can run as root when a command is preceded by ‘sudo’.
To switch to root,

$ sudo su

Add a user to ‘sudoers’,

$ sudo visudo

And add a line under ‘root ALL=(ALL:ALL) ALL’, eeplacing ‘root’ with e.g.,

ALL=(ALL:ALL) ALL

Raspberry Pi 3 Running noip2 systemd (part 9)

I find whenever the Pi has to reboot that I have to start noip2 each time with,

$ sudo noip2

This is exactly where systemd would be very useful. It would automatically restart the noip2 on boot or anytime it stops.

First we need to find where the noip2 binary is,

$ which noip2
/usr/local/bin/noip2

This is useful for finding the file that runs any command e.g

$ which ls
/bin/ls

Anyway, now we know that noip2 runs at /usr/local/bin/noip2. I would like to create a systemd service to run that binary file.

Create the file noip2.service at

/etc/systemd/system/noip2.service

[Unit]
Description=No-ip.com dynamic IP address updater
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target
Alias=noip.service

[Service]
# Start main service
ExecStart=/usr/local/bin/noip2
Restart=always
Type=forking

I obtained this file from Nathan Giesbrecht’s GitHub page.

Enable the service, so that it will start on boot.

$ sudo systemctl enable noip2.service

Then start it,

$ sudo systemctl start noip2.service

Sometimes it is necessary to also,

$ systemctl daemon-reload

Check the status of this new service with,

$ systemctl status noip2.service

To test it, reboot your Pi.

$ sudo reboot

Check that the noip2 is running automatically with,

$ ps -aux|grep noip2

The second line is I presume the process to run this command, so ignore it. The first line shows this process is running.

Raspberry Pi 3 Express Application (part 8)

Check Node and npm are installed,

$ node --version
v7.4.0
$ npm --version
4.0.5

Install Express generator globally,

$ npm install express-generator -g

Check it,

$ express --version
4.14.0

Create a new directory inside /opt/nodeserver

$ mkdir tempGraph

This will create several folders as well as the package.json file with all dependencies. Next install these dependencies,

$ sudo npm install

This automatically locates package.json in the same directory as the command is run in and installs them.

Start express with,

$ npm start

open another terminal on the same system and type,

$ curl localhost:3000

You should see same HTML,

Install nodemon to automatically restart application should it be updated, which will happen frequently during development.

$ npm install -g nodemon

To start nodemon, first top the application and be sure to be in the applications directory, then,

$ nodemon

Nodemon is for development not production.

Raspberry Pi 3 Crontab (part 7)

To run a file every 5 minutes we will use crontab,

$ crontab -e

And add the following to the bottom of that file, which opens with the ‘nano’ editor.

# m h dom mon dow command
*/5 * * * * python /opt/nodeserver/temperature.py # At every 5th minute
1 */1 * * * python /opt/nodeserver/temperatureHourlyAverages.py # At minute 1 past every hour
2 * */1 * * python /opt/nodeserver/temperatureDailyAverages.py # At minute 2 on every day-of-month

This data can be graphed using d3.js The file and graph are available at shanespi.no-ip.biz if the server is running at the time, that is. Next I want to run this with the Express backend framework as a systemd service.

Systemd could be used to run the python files instead of the crontab.

Raspberry Pi 3 DS18B20 Temperature Sensor (part 6)

Open /boot/config.txt

$ sudo nano /boot/config.txt

and add

dtoverlay=w1-gpio

to the end of the file.

Reboot the system,

$ sudo reboot

Then run,

$ sudo modprobe w1-gpio
$ sudo modprobe w1-therm

The change directory /sys/bus/w1/devices/ and list its contents,

Change directory into 28-000006c87ee2 and then ‘cat w1_slave’.

The ‘t=16125’ this a temperature reading of 16.125 degrees celcius from the temperature sensor.

Check out Adafruit’s article on reading the DS18B20.

Using Python to read the temperature sensor and output the value to a file data.json

# /opt/nodeserver/temperature.py
# crontab -e runs this file every 5 minutes
# Reads temperature from sensor
# Reads /var/www/html/data/data/json file
# Appends a reading and removes earliest reading, only ever 13 readings allowed
# Dumps data back into /var/www/html/data/data,json

import os
from decimal import Decimal
import json
import datetime
import time

now = datetime.datetime.now()

# Open the file that we viewed earlier so that python can see what is in it. Replace the serial number as before.
tfile = open("/sys/bus/w1/devices/28-000006c87ee2/w1_slave")
# Read all of the text in the file.
text = tfile.read()
# Close the file now that the text has been read.
tfile.close()
# Split the text with new lines (\n) and select the second line.
secondline = text.split("\n")[1]
# Split the line into words, referring to the spaces, and select the 10th word (counting from 0).
temperaturedata = secondline.split(" ")[9]
# The first two characters are "t=", so get rid of those and convert the temperature from a string to a number.
temperature = Decimal(temperaturedata[2:])
# Put the decimal point in the right place and display it.
temp = temperature/1000

with open('/opt/nodeserver/data/data.json', 'r') as f:
data = json.load(f)

data[0]["fiveMinReadings"].append({"temp": str(temp), "date": str(now)})

while len(data[0]["fiveMinReadings"]) > 13:
del data[0]["fiveMinReadings"][0]

with open('/opt/nodeserver/data/data.json', 'w') as f:
json.dump(data, f)

temperatureDailyAverages.py calculates an average of the last 13 values from the temperature.py file, effectively finding an average for each hour. The result is appended to data.json

# /opt/nodeserver/temperatureHourlyAverages.py
# crontab -e runs this file every hour
# Reads /var/www/html/data/data/json file
# Calculates average reading from last hour
# Appends a reading and removes earliest reading, only ever 25 readings allowed
# Dumps data back into /var/www/html/data/data.json

import os
from decimal import Decimal
import json
import datetime
import time

now = datetime.datetime.now()
with open('/opt/nodeserver/data/data.json', 'r') as f:
data = json.load(f)

sum = 0;
for i in range(0,len(data[0]["fiveMinReadings"])):
sum = sum + float(data[0]["fiveMinReadings"][i]["temp"])

average = sum/len(data[0]["fiveMinReadings"]) #this is the average reading from the previous 13 readings taken every 5 minutes
temp = round(average, 2)

data[1]["hourlyAverages"].append({"date": str(now),"temp": str(temp)})

while len(data[1]["hourlyAverages"]) > 25:
del data[1]["hourlyAverages"][0]

with open('/opt/nodeserver/html/data/data.json', 'w') as f:
json.dump(data, f)

Similarly the temperatureHourlyAverages.py

# /opt/nodeserver/temperatureHourlyAverages.py
# crontab -e runs this file every day at midnight 00:00
# Reads /var/www/html/data/data/json file
# Calculates average reading from last 24 hours
# Appends a reading and removes earliest reading, only ever 30 readings allowed
# Dumps data back into /var/www/html/data/data.json

import os
from decimal import Decimal
import json
import datetime
import time

now = datetime.datetime.now()
with open('/opt/nodeserver/data/data.json', 'r') as f:
data = json.load(f)

sum = 0;
for i in range(0,len(data[1]["hourlyAverages"])):
sum = sum + float(data[1]["hourlyAverages"][i]["temp"])

average = sum/len(data[1]["hourlyAverages"]) #this is the average reading from the previous 30 readings taken every hour
temp = round(average, 2)

data[2]["dailyAverages"].append({"date": str(now),"temp": str(temp)})

while len(data[2]["dailyAverages"]) > 30:
del data[2]["dailyAverages"][0]

with open('/opt/nodeserver/data/data.json', 'w') as f:
json.dump(data, f)

Raspberry Pi 3 Node Server & systemd (part 5)

Run a simple Node server. Create the file and path,

/opt/nodeserver/server.js

const http = require('http');
const hostname = '0.0.0.0'; // listen on all ports
const port = 8080;
http.createServer((req, res) => {
res.writeHead(200, { 'Content-Type': 'text/plain' });
res.end('Hello people, I am a server \n' + new Date().toISOString());
}).listen(port, hostname, () => {
console.log('Server running at http://' + hostname + ':' + port);
});

Run this,

$ node server.js

But this will stop running if you ctrl-c. Therefore we will use Systemd to create a service which will run this server continuously and also start it again in the case of any interruption.

Change directory to

$ cd /etc/systemd/system

Create the file nodeserver.service

$ sudo nano nodeserver.service

and add the following [Unit], [Service] and [Install] as follows,

[Unit]
Description=Node.js Example Server
#Requires=After=mysql.service # Requires the mysql service to run first

[Service]

ExecStart=/usr/bin/node /opt/nodeserver/server.js
Restart=always
RestartSec=10 # Restart service after 10 seconds if node service crashes
StandardOutput=syslog # Output to syslog
StandardError=syslog # Output to syslog
SyslogIdentifier=nodejs-example
User=pi
Group=root
Environment=NODE_ENV=production PORT=8080

[Install]
WantedBy=multi-user.target

You may need to change to ‘User’ and ‘Group’. Make sure the PORT=8080 is the same port as in server.js

Next we need to ‘enable’ and ‘start’ the service

$ sudo systemctl enable nodeserver.service
$ sudo systemctl start nodeserver.service

You can also use ‘restart’, ‘stop’ and ‘status’ to check that the service is running.

$ sudo systemctl restart nodeserver.service
$ sudo systemctl stop nodeserver.service
$ sudo systemctl status nodeserver.service

You will need to reload the daemon with

$ sudo systemctl daemon-reload

After ‘reload’ you will need ‘restart’ the service.

$ sudo systemctl restart nodeserver.service

Most of the information here is from the excellent tutorial at nodejs service with systemd
Another good source,
Deploying nodejs applications with systemd

You can check this in your browser with,

curl http://0.0.0.0:8080

If you’ve set up port-forwarding and noip.com you’ll be able to go to any browser on the internet and see the output.

Raspberry Pi 3 Static IP Address & No-ip setup (part 3)

To set up a static IP address on the Pi 3 which will remain constant inside you LAN open /etc/dhcpcd.conf

$ sudo nano dhcpcd.conf

Add to the bottom of this file,

interface eth0

static ip_address=192.168.1.30
static routers=192.168.1.1
static domain_name_servers=192.168.1.1

interface wlan0

static ip_address=192.168.1.30
static routers=192.168.1.1
static domain_name_servers=192.168.1.1

Here 192.1.68.1.30 is the static ip address I am creating.
192.1.68.1.1 is the gateway address, the address of my router from inside the network.

You will need to access your router’s settings and reserve the ip address 192.1.68.1.30 or whatever you chose, so that no other device on your network will be allocated this address.

Your router external address, that is the ip with which it is identified on the internet is usually dynamic. You can fix this using ‘no-ip’, this will allow you set it to a name instead of having to find the ever changing ip address.

Sign up for a free account at no-ip.com The free account needs to be renewed every 30 days. But this is a excellent and very useful service.

The instructions for setting up noip2 on a Raspberry Pi 3 can be found on the no-ip website.