Scenario:

I've got an RPI 4 (8GB RAM) booted up from a 1TB Samsung T3 external SSD. Unable to clone or create a working copy of the 1TB T3 SSD on another brand new 1TB Samsung T7 (newer model) SSD.

I've got Weewx running on the RPI (collecting PWS (Personal Weather Station) data and transmitting that all over the place (as an official NOAA CWOP weather station - has to be up and running 24/7 feeding accurate data)). I also have Home Assistant
running on it in a docker container, connected to and managing dozens of sensors and also even managing a home-grown alarm system for the home.

What is needed that I am struggling on achieving:

It's all tweaked and properly cooled so as not to overload the CPU or I/O (connected via ethernet and bluetooth and wifi completely turned off etc.) and I wanted to add more to it but am stopping myself and requiring a backup such that if the SSD fails I
can just shut it down, unplug the old SSD and plug in a backup SSD (a brand new samsung 1TB T7) and just boot it up to be up and running on a new SSD that is a near exactg copy no more than a few days old of the original one.

I also have a cron job on the RPI that prperly stops all the processes and reboots the thing daily a 3am, and emails me verification all that went well. I also have a program runbning on there that will turn fans on if it gets above 60 degrees C and then
shuts the fans off once it falls below 50 degrees c.

So I am treating this thing with kid gloves.

I have all the above running automatically at boot, so whenever I try to make a backup, so that the RPI is not overwhelmed on a first time bootup of a backup SSD, have changed all apps to not start up on reboot, and then rebooted the pi on the original
ssd to verify none of the weewx and home assistant processes are starting. Then I gently shut it down and the attempt a backup - as the nightmare scenario shows below:

I am unable to make a copy of the old SSD on the new one. How do I do this in a reliable manner (taking into account what I have already tried below)?

What has been tried so far (each item with the RPI temporarily turned off):

I have purchased AOMEI backupper (the free version does not allow you to clone a disk that has more than one partition) and tried to clone the old SSD to the new SSD (when both attached to a PC). The first partition clones quickly (FAT32) but the second
partition (EXT4) clone takes forever and ultimately fails, saying sometimes either the destination SSD has stopped responding, or when nearly completed (at 96%) that the partition table cannot be written because other software is using the drive (which
is not the case).

I have tried RPI Imager on a PC to put an image of the RPI onto a separate drive on the PC but that usually fails saying "data has changed" (which is not true).

I was able to use the free Win32 Disk Imager to create an image (1TB ugh) on my PC's hard drive of the SSD however. Then I have installed and used UBUNTU to shrink the .img from the 1TB to what evidently ends up being about 26GB. To verify the source
data has no issue I was able to write and unpack that 26MB image onto a 128GB Micro SD Card (it's the smallest I have) and then in linux run several utilities to correct any issues found on the file system (and it has found none). And the Micro SD Card
boots up on the RPI with no issue. YAY! BUt I can't get any further UGH!

I have then tried to unpack the .img using RPI Imager on the PC to the backup SSD but that always fails saying the destination SSD has stopped responding. So I tried AOMEI Backupper as well as AOMEI Partition Assistant to clone either the whole disk from
one SSD to the other or the whole SD Card to the backup SSD - which did not work - the clone always fails - and tried to clone one partition at a time - same story the EXT4 partition clone always fails saing it cannot create the partition table as it is
locked by other software (BS!). I had some SSD software from Samsung runing on the PC but that has al been removed and makes no difference - no software or processes having anything to do with samsung is on the PC, I even when through the PC's registry
manually and removed everything with the workd Samsung that is relaed to SSD's.

I have tried to use the RPI (when booted up on a different SSD) to use the SD Card Copier to copy the image from the running OS SSD onto the backup SSD and that always fails. I was thinking I didn't have enough power so I bought a new power supply maxing
out what can be fed to the RPI and I still have the same result. The T3 and T7 use very little power so that is not the issue. Then I also booted up the RPI from the usual SSD, stopped all the processes, the mounted both the good Micro SD Card (on a USB
adaper) and the backup SSD onto the PI, and tried to use SD Card Copier to copy the SD Card to the backup SSD but that always tells me the copy is being stopped because some of the files havce changed. Which is BS. I also tried the SD Card Copier while
the Micro SD Card is directly in the RPI itself but with the RPI booted up on the original SSD before mounting the SD Card and backup SSD onto it. Same error message, makes no difference.

So now I am thinking the new backup T7 SSD has something wrong with it ragarding bad sectors or the like. SO I did a full slow scan of that and nothing was found bad. So, from the PC I am doing a full slow reformat of the T7. It has been running for 4
days now and is at 74% (slow but steady progress). Then I was going to try to write to it (the 1TB T7) again either from the Micro SD Card or the original SSD, I have not decided which yet...?

Last item is, I was going to once I have the second SSD finally created, I was going to boot it up on the RPI by itself and then have it start up with Weewx and Home Assistant processes make sure it works and have them then come up on restart. Then if
worse comes to worse and it's impossible to clone this SSD every now and again, to just livce with both SSD's working (onloy one at a time of course) and every time I make a change to one on the RPI I was going to just boot up the other SSD and make the
same change there. If I do it that way instead of redoing nightmare backups, then I was going to figure out which specific DB (and related index) files have the history on them it and just copy those files from one SSD to the other from time to time.

So once the format on the T7 is completed (which might or might not solve my nightmare issue) - how should I try the restore to it?

Anyone have any ideas ? I was going to look into rsnapshot or clonezilla (I did try clonezilla but didn't even get it to run, I'm a noob at linux (but used to design trading systems for wall street in C++ etc.)) or tuxboot - UGH!

Anyone have any ideas before I decide to defenestrate or shoot myself in the head (just kidding)?

--- SoupGate-Win32 v1.05
* Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)

On 2022-05-17, Kruse Ludington <rkludington@gmail.com> wrote:

Scenario:

I've got an RPI 4 (8GB RAM) booted up from a 1TB Samsung T3 external
SSD. Unable to clone or create a working copy of the 1TB T3 SSD on
another brand new 1TB Samsung T7 (newer model) SSD.

Once a week I use SD Card Copier to clone my SSD to a flash drive. I
make sure to check the New Partition UUIDs box before performing the
copy. What I end up with is an emergency bootable flash drive. This
works every time.

Make sure your destination SSD is formatted MBR/FAT before attempting
the clone.

--
chrispam1@me.com is a filtered spam magnet. Email replies may be lost.
You're better off replying to this newsgroup.

--- SoupGate-Win32 v1.05
* Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)

On a sunny day (Mon, 16 May 2022 18:19:11 -0700 (PDT)) it happened Kruse Ludington <rkludington@gmail.com> wrote in <62132ea3-9f41-46cb-a3b9-964ef6b02b53n@googlegroups.com>:

Scenario:

I've got an RPI 4 (8GB RAM) booted up from a 1TB Samsung T3 external SSD. Unable
to clone or create a working copy of the 1TB T3 SSD on another brand new
1TB Samsung T7 (newer model) SSD.

....

Anyone
have any ideas before I decide to defenestrate or shoot myself in the
head (just kidding)?

Yes, I recently found internet is getting a bit stressed with so many things blocked, sucking MB web sites, etc.
Without internet would life be better?

The backups:
I do not have SSDs but large 3.5 TB harddisks.

[I] Never backup from a running system

as you pointed out I simply copy what changed and was important to the other disk on the other raspi.
For backups of Pi SDcards I power down the Pi, take the card out, put it in the laptop,
and link to the other Pi with the other 3.5TB disk (but I also have a USB card reader I could use).

I tried scp -p /dev/raspi_card IP_OTHER_PI:/mnt/sda2/backups/my_pi4_card.img scp then complained /dev/raspi_card is not a file....
(all in UNIX is a file so whats that about??)
(rapi_card stands for sdc or sdd depending whatever else you have hanging from the laptop)
and make sure it is NOT MOUNTED, some system like to mount things automatically I think.

OK, so then netcat to the rescue!
on the destination Pi I ran:
netcat -l -p 1029 > /mnt/sda2/backups/my_pi4_card.img
here netcat listens on port 1029 and sends output /mnt/sda2/backups/my_pi4_card.img

And on the sending site (laptop) did:
cat /dev/raspi_card | netcat -a IP_OTHER_PI -p 1029
sends whole card image to IP IP_OTHER_PI
took a while via the LAN..
There are different versions of netcat around with different arguments to stop transfer on EOF, so check man netcat.

Anyways, if you want easier access to files later, then zip the partitions mount /dev/raspi_partition1 /mnt/raspi_partition1
mount /dev/raspi_partition2 /mnt/raspi_partition2

tar -zcvf raspi_partition1.tgz /mnt/raspi_partition1
tar -zcvf raspi_partition2.tgz /mnt/raspi_partition2
then copy the compressed partitions in the normal way (scp should work :-) ) That allows you to unzip it later and get a specific file.

Having the backup thing saved me recently from disaster,

I also backup Raspi SDcards to other SDcards (all Samsung here) so I can just swap cards if something strange happens.
There is a bit more to it, but OK.



As to data transfer via internet, maybe using a drone and MicroSDcards is faster and cheaper,
couple of TB ... over miles.. :-)
http://panteltje.com/panteltje/quadcopter/index.html
or posting pigeons with SDcards..

Anyways after WW3 nukin likely none of the electronics shit will still work.
I do keep some optical backups too though, among those M-Discs, say for the archaeologists..
http://panteltje.com/pub/CD_box_binnenkant_IXIMG_0549.JPG

--- SoupGate-Win32 v1.05
* Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)

On 17/05/2022 02:19, Kruse Ludington wrote:

Anyone have any ideas ? I was going to look into rsnapshot or clonezilla (I did try clonezilla but didn't even get it to run, I'm a noob at linux (but used to design trading systems for wall street in C++ etc.)) or tuxboot - UGH!

Anyone have any ideas before I decide to defenestrate or shoot myself in the head (just kidding)?

The way I do this oustide of a pi environment is simply to not attempt
to clone the whole drive of a machine running off that drive. the boot environment is not normally accessible to the running machines OS.
Fortunately the boot environment is stable, and does not vary much - if
at all.
So my suggestion would be to install a bare OS/boot partition on the
backup disk, and then rsync the data to it from the working one.

In my desktop/laptop setups I simply now have realised that upgrading to
a major new operating system is a chance to rid the PC of the cruft that accumulates. So a fresh install on a new hard drive and then replace the programs I find I still need :-)

--
“The fundamental cause of the trouble in the modern world today is that
the stupid are cocksure while the intelligent are full of doubt."

- Bertrand Russell

--- SoupGate-Win32 v1.05
* Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)

On 17/05/2022 11:27, The Natural Philosopher wrote:

On 17/05/2022 02:19, Kruse Ludington wrote:

Anyone have any ideas ? I was going to look into rsnapshot or
clonezilla (I did try clonezilla but didn't even get it to run, I'm a
noob at linux (but used to design trading systems for wall street in
C++ etc.)) or tuxboot - UGH!

Anyone have any ideas before I decide to defenestrate or shoot myself
in the head (just kidding)?

The way I do this oustide of a pi environment is simply to not attempt
to clone the whole drive of a machine running off that drive. the boot environment is not normally accessible to the running machines OS. Fortunately the boot environment is stable, and does not vary much -  if
at all.
So my suggestion would be to install a bare OS/boot partition on the
backup disk,  and then rsync the data to it from the working one.

In my desktop/laptop setups I simply now have realised that upgrading to
a major new operating system is a chance to rid the PC of the cruft that accumulates. So a fresh install on a new hard drive and then replace the programs I find I still need :-)

For the rPi just use the SDcard for boot device. I'm not convinced
booting off an SSD is normally justified.

Leaving the OS reasonably untouched on an SDcard and working using
Docker Containers on the SSD works well. Just back up the Docker Data containers, save Docker config in git.

Reinstalling the OS + Docker containers is then trivial, so doesn't need backing up.

OK, reinstall is not trivial, but can be done in 11 easy steps :o). (11
is a joke, not literal.) It takes an hour or two, but I like to do it occasionally to refresh my memory.

I understand Ubuntu is now using snaps. They sound cool, and I may look
at using them the next time my rPi home server goes tits up.

--- SoupGate-Win32 v1.05
* Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)

On Mon, 16 May 2022 18:19:11 -0700 (PDT), Kruse Ludington wrote:

Scenario:

I've got an RPI 4 (8GB RAM) booted up from a 1TB Samsung T3 external
SSD. Unable to clone or create a working copy of the 1TB T3 SSD on
another brand new 1TB Samsung T7 (newer model) SSD.

I also have a cron job on the RPI that prperly stops all the processes
and reboots the thing daily a 3am, and emails me verification all that
went well. I also have a program runbning on there that will turn fans
on if it gets above 60 degrees C and then shuts the fans off once it
falls below 50 degrees c.

Here's what I'd do (assuming I wanted to preserve the partitioning of the master SSD:

1) Get a cycle of at least two good quality 8GB SSD (cards or SSD -
doesn't matter which, but SD cards are cheaper: I like Sandisk SDs . Give
each a distinct name and some sort of physical storage so you can easily
know which was the most recent backup - a block of wood with saw cuts
would be fine if you're backing up to SD cards. In the following I assume you're using SD cards. Have a safe place to keep them (personal firesafe,
etc).

2) Extend your reboot script to:
a - run 'fsck' on the whole ssd: this gives advance warning of media
failures
b - make a full copy of your master SSD to the backup SD using 'dd' to
make the backup
c - now run the remainder of your reboot script.

Using this sequence makes sure you backup is clean and that you won't ry
to reboot from a corrupted filing system.

3) If you're not already running 'logwatch', enable it. It defaults to
running once a day and defaults to sending a system state report to
'root', but its trivial to install a mail server (Postfix is great) and configure logwatch so its daily report gets sent to your usual mail
stream. Logwatch runs an extendable list of subsystem reports, so you
might want to add one to report exceptions thrown by your weather service.
"man logwatch" containd pointers for configuring and customising logwatch.

Now all you need to do is once a day read the logwatch report to check
that all is well, remove the last overnight backup card from the Pi, put
in the offline SD card rack and replace it with the oldest backup, which
will get overwritten early tomorrow.

4) You could use rsnapshot to do the backup, but, configuring it to do
exactly what you want, assuming you want to keep the partition contents separate will be a bit harder than using 'dd' - IIRC rsnapshot tends to
build a single directory structure while 'dd' understands partitioning
and will preserve your partition structure on the backup device.

--- SoupGate-Win32 v1.05
* Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)

On 17/05/2022 02:19, Kruse Ludington wrote:

Scenario:

I've got an RPI 4 (8GB RAM) booted up from a 1TB Samsung T3 external SSD. Unable to clone or create a working copy of the 1TB T3 SSD on another brand new 1TB Samsung T7 (newer model) SSD.

It's all tweaked and properly cooled so as not to overload the CPU or I/O (connected via ethernet and bluetooth and wifi completely turned off etc.)

It's a Pi 4B, it's capable of doing a lot, what you want wouldn't even
tax a Pi 3B, so don't worry.

and I wanted to add more to it but am stopping myself and requiring a backup such that if the SSD fails I can just shut it down, unplug the old SSD and plug in a backup SSD (a brand new samsung 1TB T7) and just boot it up to be up and running on a new

SSD that is a near exactg copy no more than a few days old of the original one.

That's why I do with all my Pi's.

I also have a cron job on the RPI that prperly stops all the processes and reboots the thing daily a 3am, and emails me verification all that went well.

That's unnecessary, you just increase the risk failure by unnecessary rebooting. It better to enable the hardware watchdog which will make
sure that it automatically reboots if the OS hangs or (optionally) the
network is unreachable.

I also have a program runbning on there that will turn fans on if it gets above 60 degrees C and then shuts the fans off once it falls below 50 degrees c.

If it's not in an environment where the noise is an issue, just leave
the fan on all the time, less to go wrong.

So I am treating this thing with kid gloves.

They are tough, my Pi original 256MB is still refusing to die after 190
years of use.

I have all the above running automatically at boot, so whenever I try to make a backup, so that the RPI is not overwhelmed on a first time bootup of a backup SSD, have changed all apps to not start up on reboot, and then rebooted the pi on the original

ssd to verify none of the weewx and home assistant processes are starting. Then I gently shut it down and the attempt a backup - as the nightmare scenario shows below:

I am unable to make a copy of the old SSD on the new one. How do I do this in a reliable manner (taking into account what I have already tried below)?

The easiest way to do it is you have another Linux machine, an old PC or another Pi will do. First using the other machine clone the Pi's SDD to
the backup SSD using the dd command. Share the backup SSD over NFS so
the Pi can mount it. Set up a cron job on the Pi to do a differential
backup to the other SSD every night.

If those terms are not familiar, there are lots of tutorials on how to
use dd, NFS, cron and rsync.

What that will do is give you a backup SSD which is at most 24 hours
old, and ready plug straight in to the Pi should the primary fail.

I assume you are booting direct from the SSD, as if not the SD card will probably fail long before the SSD, bit that can be backed up in exactly
the same way.

---druck

--- SoupGate-Win32 v1.05
* Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)