• backing up a PI SD card..

    From The Natural Philosopher@3:770/3 to All on Sun Sep 17 10:19:37 2023
    It occurs to me that essentially a copy of the contents of the two PI partitions would be enough, together with an install of Raspios,
    followed by a copy back to the fresh SD, to clone a setup.

    It seems too easy. What am I missing?


    --
    “It is not the truth of Marxism that explains the willingness of intellectuals to believe it, but the power that it confers on
    intellectuals, in their attempts to control the world. And since...it is
    futile to reason someone out of a thing that he was not reasoned into,
    we can conclude that Marxism owes its remarkable power to survive every criticism to the fact that it is not a truth-directed but a
    power-directed system of thought.”
    Sir Roger Scruton

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Knute Johnson@3:770/3 to The Natural Philosopher on Sun Sep 17 15:19:05 2023
    On 9/17/23 04:19, The Natural Philosopher wrote:

    It occurs to me that essentially a copy of the contents of the two PI partitions would be enough, together with an install of Raspios,
    followed by a copy back to the fresh SD, to clone a setup.

    It seems too easy. What am I missing?



    Raspberry -> Accessories -> SD Card Copier

    --

    Knute Johnson

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to Knute Johnson on Mon Sep 18 11:08:10 2023
    On 17/09/2023 21:19, Knute Johnson wrote:
    On 9/17/23 04:19, The Natural Philosopher wrote:

    It occurs to me that essentially a copy of the contents of the two PI
    partitions would be enough, together with an install of Raspios,
    followed by a copy back to the fresh SD, to clone a setup.

    It seems too easy. What am I missing?



    Raspberry -> Accessories -> SD Card Copier

    What has a raspberry to do with this?

    --
    “People believe certain stories because everyone important tells them,
    and people tell those stories because everyone important believes them.
    Indeed, when a conventional wisdom is at its fullest strength, one’s agreement with that conventional wisdom becomes almost a litmus test of
    one’s suitability to be taken seriously.”

    Paul Krugman

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Newyana2@3:770/3 to The Natural Philosopher on Mon Sep 18 09:03:00 2023
    "The Natural Philosopher" <tnp@invalid.invalid> wrote

    | > Raspberry -> Accessories -> SD Card Copier
    | >
    | What has a raspberry to do with this?
    |
    Start menu. The raspberry icon in lower left.
    You didn't say which Pi you have, so you may not
    have that icon or the copier software.

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to All on Mon Sep 18 14:33:38 2023
    On 18/09/2023 14:03, Newyana2 wrote:
    "The Natural Philosopher" <tnp@invalid.invalid> wrote

    | > Raspberry -> Accessories -> SD Card Copier
    | >
    | What has a raspberry to do with this?
    |
    Start menu. The raspberry icon in lower left.

    ??? Icon? On a headless Pi Zero?

    You didn't say which Pi you have, so you may not
    have that icon or the copier software.

    Pi Zero



    --
    The lifetime of any political organisation is about three years before
    its been subverted by the people it tried to warn you about.

    Anon.

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Newyana2@3:770/3 to The Natural Philosopher on Mon Sep 18 22:15:47 2023
    "The Natural Philosopher" <tnp@invalid.invalid> wrote

    | > |
    | > Start menu. The raspberry icon in lower left.
    |
    | ??? Icon? On a headless Pi Zero?
    |
    Ah. You might have said that in the first place. It
    doesn't make much sense to ask for help and then
    be difficult with people who try to help. I gather you
    only posted to argue.

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to All on Tue Sep 19 08:15:03 2023
    On 19/09/2023 03:15, Newyana2 wrote:
    "The Natural Philosopher" <tnp@invalid.invalid> wrote

    | > |
    | > Start menu. The raspberry icon in lower left.
    |
    | ??? Icon? On a headless Pi Zero?
    |
    Ah. You might have said that in the first place. It
    doesn't make much sense to ask for help and then
    be difficult with people who try to help. I gather you
    only posted to argue.


    No. I am trying to establish whether one can, in essence, make a
    bootable PI SD card by taking a raw card, creating the two partitions on
    it and loading it up with files, or whether there is the equivalent of a
    boot sector which would mandate the use of e.g. dd to create it

    Naturally all the documentation assumes a windows numpty and shows you
    how to stamp the SD card with an image.

    You assumed I wanted to clone the card. I don't. I want to back one up
    without using DD because, as the concurrent thread has identified, the
    PI expands the file system to the whole card and that makes for a big image. I'd rather back it up as files if possible.

    If at some future date I need to recreate it, instead of dd-ing the
    image, I would prefer to create the two partitions and populate them
    from the tarball.



    --
    There is nothing a fleet of dispatchable nuclear power plants cannot do
    that cannot be done worse and more expensively and with higher carbon
    emissions and more adverse environmental impact by adding intermittent renewable energy.

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Computer Nerd Kev@3:770/3 to The Natural Philosopher on Tue Sep 19 19:11:34 2023
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    No. I am trying to establish whether one can, in essence, make a
    bootable PI SD card by taking a raw card, creating the two partitions on
    it and loading it up with files,

    Yes. In fact if you set the OS up from scratch you don't even need
    two partitions, just one FAT32 parition with the boot files
    required by the Pi.

    I set up a Linux system to boot off the FAT32 parition then run in
    a tmpfs. So instead of an image file there's just a ZIP archive
    which needs to be unpacked to the root of a FAT32-formatted SD
    card. No image writer tool required, and no worries about partition
    sizes.

    The Pi doesn't need anything like a boot sector. The firmware is
    smart enough to find the boot files on the filesystem, provided
    it's FAT32.

    --
    __ __
    #_ < |\| |< _#

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to Computer Nerd Kev on Tue Sep 19 10:35:23 2023
    On 19/09/2023 10:11, Computer Nerd Kev wrote:
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    No. I am trying to establish whether one can, in essence, make a
    bootable PI SD card by taking a raw card, creating the two partitions on
    it and loading it up with files,

    Yes. In fact if you set the OS up from scratch you don't even need
    two partitions, just one FAT32 parition with the boot files
    required by the Pi.

    I set up a Linux system to boot off the FAT32 parition then run in
    a tmpfs. So instead of an image file there's just a ZIP archive
    which needs to be unpacked to the root of a FAT32-formatted SD
    card. No image writer tool required, and no worries about partition
    sizes.

    The Pi doesn't need anything like a boot sector. The firmware is
    smart enough to find the boot files on the filesystem, provided
    it's FAT32.


    I suppose that needs to be the first partition?
    The other question why always gets me on backups is how many of the
    directories in the root partition can safely be ignored?

    /sys, /proc, /run would seem to be created by the machine at boot time,
    and shouldn't be created as subdirs of the root partition...or should
    the directories be created, but empty?

    So putatively one might back up everything in / except /run, /proc &
    /sys, including /boot, plonk a shiny new SD card in a host computer,
    create a VFAT partition of say 256MB, and the rest as ext4, untar the
    archive into the ext4 partition and move the entire contents of /boot
    onto the VFAT, shove the card in a pi , apply power, and Robert should
    be a relative?

    The vfat partition will get mounted on the (now empty) /boot mount
    point, and everything else will be where it ought to be, and /run /sys
    and /proc will get generated by the boot process?

    What about /dev? Is that dynamically created or not?





    --
    "Anyone who believes that the laws of physics are mere social
    conventions is invited to try transgressing those conventions from the
    windows of my apartment. (I live on the twenty-first floor.) "

    Alan Sokal

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Martin Gregorie@3:770/3 to The Natural Philosopher on Tue Sep 19 19:05:08 2023
    On Tue, 19 Sep 2023 08:15:03 +0100, The Natural Philosopher wrote:

    On 19/09/2023 03:15, Newyana2 wrote:
    "The Natural Philosopher" <tnp@invalid.invalid> wrote

    | > |
    | > Start menu. The raspberry icon in lower left. |
    | ??? Icon? On a headless Pi Zero?
    |
    Ah. You might have said that in the first place. It
    doesn't make much sense to ask for help and then be difficult with
    people who try to help. I gather you only posted to argue.


    No. I am trying to establish whether one can, in essence, make a
    bootable PI SD card by taking a raw card, creating the two partitions on
    it and loading it up with files, or whether there is the equivalent of a
    boot sector which would mandate the use of e.g. dd to create it

    Naturally all the documentation assumes a windows numpty and shows you
    how to stamp the SD card with an image.

    You assumed I wanted to clone the card. I don't. I want to back one up without using DD because, as the concurrent thread has identified, the
    PI expands the file system to the whole card and that makes for a big
    image.
    I'd rather back it up as files if possible.

    If at some future date I need to recreate it, instead of dd-ing the
    image, I would prefer to create the two partitions and populate them
    from the tarball.

    Take a look at 'rsync' - its my go-to tool for backing up complete Linux
    filing systems, specific partitions or lists of files and partitions to as
    a named directory structure on a backup disk.

    It is designed make the backup directory structure identical to the
    current list of files and directories you ask it to back up, IOW, if you
    rerun the sa,me command a week later it will remove any files and
    directories from the previous backup session, replace any files and
    directories that have been changed and add any new files and directories.

    Consequently, subsequent runs will usually be much faster than the initial backup.

    Want to keep older backups? Just use a cycle of backup files. SD-cards or USB-connected disks: I back up entire Linux file systems to a cycle of two
    1TB WD Essentials portable disks, and do the backup immediately before the weekly Linux software update.One 1TB disk is enough to hold complete
    backup cycles for both my house server this laptop and one or more RPi
    backups.

    Retrieving whole partition contents, directories and/or files from these backups is simple: simply mount the backup volume and copy anything you've
    lost back to where it should be on your computer's filing system.


    rsync manuals and software are in included in most mainstream Linux
    distros.

    HTH

    --

    Martin | martin at
    Gregorie | gregorie dot org

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Martin Gregorie@3:770/3 to The Natural Philosopher on Tue Sep 19 19:13:54 2023
    On Tue, 19 Sep 2023 10:35:23 +0100, The Natural Philosopher wrote:

    On 19/09/2023 10:11, Computer Nerd Kev wrote:
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    No. I am trying to establish whether one can, in essence, make a
    bootable PI SD card by taking a raw card, creating the two partitions
    on it and loading it up with files,

    Yes. In fact if you set the OS up from scratch you don't even need two
    partitions, just one FAT32 parition with the boot files required by the
    Pi.

    I set up a Linux system to boot off the FAT32 parition then run in a
    tmpfs. So instead of an image file there's just a ZIP archive which
    needs to be unpacked to the root of a FAT32-formatted SD card. No image
    writer tool required, and no worries about partition sizes.

    The Pi doesn't need anything like a boot sector. The firmware is smart
    enough to find the boot files on the filesystem, provided it's FAT32.


    I suppose that needs to be the first partition?

    Nope: just whatever partition the RPi installer configured to be the boot partition.

    So putatively one might back up everything in / except /run, /proc &
    /sys, including /boot, plonk a shiny new SD card in a host computer,
    create a VFAT partition of say 256MB, and the rest as ext4, untar the
    archive into the ext4 partition and move the entire contents of /boot
    onto the VFAT, shove the card in a pi , apply power, and Robert should
    be a relative?

    Just tell dd to copy everything from the source SD card to the new one. If
    the latter card is bigger,use parted to extend the new ext4 partition and
    to format the new sectors.



    --

    Martin | martin at
    Gregorie | gregorie dot org

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From druck@3:770/3 to Martin Gregorie on Tue Sep 19 21:37:53 2023
    On 19/09/2023 20:05, Martin Gregorie wrote:
    On Tue, 19 Sep 2023 08:15:03 +0100, The Natural Philosopher wrote:
    No. I am trying to establish whether one can, in essence, make a
    bootable PI SD card by taking a raw card, creating the two partitions on
    it and loading it up with files, or whether there is the equivalent of a
    boot sector which would mandate the use of e.g. dd to create it

    Take a look at 'rsync' - its my go-to tool for backing up complete Linux filing systems, specific partitions or lists of files and partitions to as
    a named directory structure on a backup disk.

    It is designed make the backup directory structure identical to the
    current list of files and directories you ask it to back up, IOW, if you rerun the sa,me command a week later it will remove any files and
    directories from the previous backup session, replace any files and directories that have been changed and add any new files and directories.

    Consequently, subsequent runs will usually be much faster than the initial backup.

    Here's the rsync options I used to backup the partitions, firstly the
    FAT partition which accounts for the lower resolution of timestamps

    rsync -vax --stats --modify-window=2 --delete /boot/ <boot_backup_dir>

    And the main partition with the exclusions to prevent pseudo filing
    systems and temporary stuff being included when backing up live systems,
    some of it shouldn't be necessary but turns out is needed

    rsync -vaxHAX --stats --numeric-ids --delete --delete-excluded
    --exclude=/tmp/*
    --exclude=/run/*
    --include=/var/lock
    --exclude=/var/run
    --exclude=/var/swap
    --exclude=/var/cache/*
    --exclude=/var/tmp/*
    --exclude=/var/lib/machines/*/var/cache/
    --exclude=/var/lib/mlocate/*
    --exclude=/var/lib/nginx/fastcgi/*
    --exclude=/var/log/journal/*
    --exclude=/var/log/Xorg.*
    --exclude=/lost+found/*
    --exclude=*/.cache
    --exclude=*/__pycache__
    --exclude=*/chromium/hyphen-data/
    --exclude=*/.config/chromium/*Cache*/*
    / <root_backup_dir>

    ---druck

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Theo@3:770/3 to The Natural Philosopher on Tue Sep 19 21:55:24 2023
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    /sys, /proc, /run would seem to be created by the machine at boot time,
    and shouldn't be created as subdirs of the root partition...or should
    the directories be created, but empty?

    /sys, /proc and /dev are filesystems which are mounted at boot time. You
    need the empty directories to act as mount points, because Unix mounts filesystems on top of a directory that already exists, switching out
    whatever is there.

    It seems that some distros have /var/run and point a symlink from /run to
    it, so it doesn't need a mount point created for them. They then mount a
    tmpfs at /var/run (so /var/run is an empty directory that needs to exist to
    be mounted on top of)

    So putatively one might back up everything in / except /run, /proc &
    /sys, including /boot, plonk a shiny new SD card in a host computer,
    create a VFAT partition of say 256MB, and the rest as ext4, untar the
    archive into the ext4 partition and move the entire contents of /boot
    onto the VFAT, shove the card in a pi , apply power, and Robert should
    be a relative?

    I think that will probably work.

    The vfat partition will get mounted on the (now empty) /boot mount
    point, and everything else will be where it ought to be, and /run /sys
    and /proc will get generated by the boot process?

    One thing to watch that distros' /etc/fstab often mounts partitions based
    on a UUID. So /boot on one SD will have a different UUID to another, and
    the mount of /boot will fail. It'll still find the kernel from /boot, but
    any time later anything wants to load something from /boot it may fail
    because the mount failed. You could change the device name in /etc/fstab to
    be /dev/mmcblk0p1 or similar to be sure it's the same every time.
    (the Pi has a single SD slot so no chance of mixing it up with another)

    'blkid' will show your partition devices with their UUIDs.

    What about /dev? Is that dynamically created or not?

    A dynamic filesystem mounted to a static pre-existing mount point.

    Theo

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Computer Nerd Kev@3:770/3 to The Natural Philosopher on Wed Sep 20 08:19:12 2023
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    On 19/09/2023 10:11, Computer Nerd Kev wrote:
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    The Pi doesn't need anything like a boot sector. The firmware is
    smart enough to find the boot files on the filesystem, provided
    it's FAT32.

    I suppose that needs to be the first partition?

    "It is no longer necessary for the first partition to be the FAT
    partition, as the MSD boot will continue to search for a FAT
    partition beyond the first one."
    https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#boot-sequence

    The other question why always gets me on backups is how many of the directories in the root partition can safely be ignored?

    /sys, /proc, /run would seem to be created by the machine at boot time,
    and shouldn't be created as subdirs of the root partition...or should
    the directories be created, but empty?

    /sys and /proc can be created empty, because they're mountpoints.
    However I don't bother excluding them from backups personally.

    So putatively one might back up everything in / except /run, /proc &
    /sys, including /boot, plonk a shiny new SD card in a host computer,
    create a VFAT partition of say 256MB, and the rest as ext4, untar the
    archive into the ext4 partition and move the entire contents of /boot
    onto the VFAT, shove the card in a pi , apply power, and Robert should
    be a relative?

    The vfat partition will get mounted on the (now empty) /boot mount
    point, and everything else will be where it ought to be, and /run /sys
    and /proc will get generated by the boot process?

    Yes. Mounting the FAT32 partition at /boot is specific behaviour
    for RPi OS / Debian. Linux itself doesn't care if that partition
    gets mounted at all - by the time the kernel is loaded its job has
    been done, but Debian probably expects it there for configuration
    and upgrades.

    What about /dev? Is that dynamically created or not?

    The device files need backing up, or else something might have to
    create them with mknod. It contains more mountpoints though, so
    you can look at the list displayed by "mount" and exclude those
    sub-directories of /dev if you like. Again, I don't bother to do
    that.

    --
    __ __
    #_ < |\| |< _#

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to Martin Gregorie on Wed Sep 20 04:37:08 2023
    On 19/09/2023 20:13, Martin Gregorie wrote:
    On Tue, 19 Sep 2023 10:35:23 +0100, The Natural Philosopher wrote:

    On 19/09/2023 10:11, Computer Nerd Kev wrote:
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    No. I am trying to establish whether one can, in essence, make a
    bootable PI SD card by taking a raw card, creating the two partitions
    on it and loading it up with files,

    Yes. In fact if you set the OS up from scratch you don't even need two
    partitions, just one FAT32 parition with the boot files required by the
    Pi.

    I set up a Linux system to boot off the FAT32 parition then run in a
    tmpfs. So instead of an image file there's just a ZIP archive which
    needs to be unpacked to the root of a FAT32-formatted SD card. No image
    writer tool required, and no worries about partition sizes.

    The Pi doesn't need anything like a boot sector. The firmware is smart
    enough to find the boot files on the filesystem, provided it's FAT32.


    I suppose that needs to be the first partition?

    Nope: just whatever partition the RPi installer configured to be the boot partition.

    I think we are not on the same hymn sheet.. The point is to NOT use a
    'pi installer'


    So putatively one might back up everything in / except /run, /proc &
    /sys, including /boot, plonk a shiny new SD card in a host computer,
    create a VFAT partition of say 256MB, and the rest as ext4, untar the
    archive into the ext4 partition and move the entire contents of /boot
    onto the VFAT, shove the card in a pi , apply power, and Robert should
    be a relative?

    Just tell dd to copy everything from the source SD card to the new one. If the latter card is bigger,use parted to extend the new ext4 partition and
    to format the new sectors.

    That is precisely what I do not want to do, and what this thread is all
    about, because DD is simply too blunt an instrument.





    --
    "And if the blind lead the blind, both shall fall into the ditch".

    Gospel of St. Mathew 15:14

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to Martin Gregorie on Wed Sep 20 04:35:18 2023
    On 19/09/2023 20:05, Martin Gregorie wrote:
    On Tue, 19 Sep 2023 08:15:03 +0100, The Natural Philosopher wrote:

    On 19/09/2023 03:15, Newyana2 wrote:
    "The Natural Philosopher" <tnp@invalid.invalid> wrote

    | > |
    | > Start menu. The raspberry icon in lower left. |
    | ??? Icon? On a headless Pi Zero?
    |
    Ah. You might have said that in the first place. It
    doesn't make much sense to ask for help and then be difficult with
    people who try to help. I gather you only posted to argue.


    No. I am trying to establish whether one can, in essence, make a
    bootable PI SD card by taking a raw card, creating the two partitions on
    it and loading it up with files, or whether there is the equivalent of a
    boot sector which would mandate the use of e.g. dd to create it

    Naturally all the documentation assumes a windows numpty and shows you
    how to stamp the SD card with an image.

    You assumed I wanted to clone the card. I don't. I want to back one up
    without using DD because, as the concurrent thread has identified, the
    PI expands the file system to the whole card and that makes for a big
    image.
    I'd rather back it up as files if possible.

    If at some future date I need to recreate it, instead of dd-ing the
    image, I would prefer to create the two partitions and populate them
    from the tarball.

    Take a look at 'rsync' - its my go-to tool for backing up complete Linux filing systems, specific partitions or lists of files and partitions to as
    a named directory structure on a backup disk.

    Been using it for years to do nightly backups of all my data.

    It is designed make the backup directory structure identical to the
    current list of files and directories you ask it to back up, IOW, if you rerun the sa,me command a week later it will remove any files and
    directories from the previous backup session, replace any files and directories that have been changed and add any new files and directories.

    Consequently, subsequent runs will usually be much faster than the initial backup.

    They are. When I replaced my backup drive because it went faulty the
    machine ran all night filling it up :-)

    Mostly its about a 3 minute run.

    However this isn't about backup, so much as recovering from an SD card
    failure.

    No data on this machine ever changes - unless its reprogrammed at a user
    level, and that is easy to replicate in half an hour

    I probably wouldn't ever bother to update or upgrade its operating
    system either.
    It will sit there, 'bits in silicon' , just doing the one thing it was
    designed to do until I move into a gaga home because I cant remember who
    I am :-)




    --
    If I had all the money I've spent on drink...
    ..I'd spend it on drink.

    Sir Henry (at Rawlinson's End)

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to druck on Wed Sep 20 04:39:40 2023
    On 19/09/2023 21:37, druck wrote:
    On 19/09/2023 20:05, Martin Gregorie wrote:
    On Tue, 19 Sep 2023 08:15:03 +0100, The Natural Philosopher wrote:
    No.  I am trying to establish whether one can, in essence, make a
    bootable PI SD card by taking a raw card, creating the two partitions on >>> it and loading it up with files, or whether there is the equivalent of a >>> boot sector which would mandate the use of e.g. dd to create it

    Take a look at 'rsync' - its my go-to tool for backing up complete Linux
    filing systems, specific partitions or lists of files and partitions
    to as
    a named directory structure on a backup disk.

    It is designed make the backup directory structure identical to the
    current list of files and directories you ask it to back up, IOW, if you
    rerun the sa,me command a week later it will remove any files and
    directories from the previous backup session, replace any files and
    directories that have been changed and add any new files and directories.

    Consequently, subsequent runs will usually be much faster than the
    initial
    backup.

    Here's the rsync options I used to backup the partitions, firstly the
    FAT partition which accounts for the lower resolution of timestamps

    rsync -vax --stats --modify-window=2 --delete /boot/ <boot_backup_dir>

    And the main partition with the exclusions to prevent pseudo filing
    systems and temporary stuff being included when backing up live systems,
    some of it shouldn't be necessary but turns out is needed

    rsync -vaxHAX --stats --numeric-ids --delete --delete-excluded --exclude=/tmp/*
    --exclude=/run/*
    --include=/var/lock
    --exclude=/var/run
    --exclude=/var/swap
    --exclude=/var/cache/*
    --exclude=/var/tmp/*
    --exclude=/var/lib/machines/*/var/cache/
    --exclude=/var/lib/mlocate/*
    --exclude=/var/lib/nginx/fastcgi/*
    --exclude=/var/log/journal/*
    --exclude=/var/log/Xorg.*
    --exclude=/lost+found/*
    --exclude=*/.cache
    --exclude=*/__pycache__
    --exclude=*/chromium/hyphen-data/
    --exclude=*/.config/chromium/*Cache*/*
    / <root_backup_dir>

    ---druck

    Looks very like what my *86 big main server has..but that is a handy
    crib and thank you very much for it.

    I'll have to look at comms between my main backup machine and the pi,
    but that may after all be the simplest solution



    --
    To ban Christmas, simply give turkeys the vote.

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to Theo on Wed Sep 20 04:48:42 2023
    On 19/09/2023 21:55, Theo wrote:
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    /sys, /proc, /run would seem to be created by the machine at boot time,
    and shouldn't be created as subdirs of the root partition...or should
    the directories be created, but empty?

    /sys, /proc and /dev are filesystems which are mounted at boot time. You need the empty directories to act as mount points, because Unix mounts filesystems on top of a directory that already exists, switching out
    whatever is there.

    It seems that some distros have /var/run and point a symlink from /run to
    it, so it doesn't need a mount point created for them. They then mount a tmpfs at /var/run (so /var/run is an empty directory that needs to exist to be mounted on top of)

    So putatively one might back up everything in / except /run, /proc &
    /sys, including /boot, plonk a shiny new SD card in a host computer,
    create a VFAT partition of say 256MB, and the rest as ext4, untar the
    archive into the ext4 partition and move the entire contents of /boot
    onto the VFAT, shove the card in a pi , apply power, and Robert should
    be a relative?

    I think that will probably work.

    Good...Well at least we dont have to flirt with 'secure boot' and all
    the other trash that comes with a *86 board.
    Nice old school 'put this here, and the boot loader will find it and
    boot it'

    The vfat partition will get mounted on the (now empty) /boot mount
    point, and everything else will be where it ought to be, and /run /sys
    and /proc will get generated by the boot process?

    One thing to watch that distros' /etc/fstab often mounts partitions based
    on a UUID. So /boot on one SD will have a different UUID to another, and
    the mount of /boot will fail. It'll still find the kernel from /boot, but any time later anything wants to load something from /boot it may fail because the mount failed. You could change the device name in /etc/fstab to be /dev/mmcblk0p1 or similar to be sure it's the same every time.
    (the Pi has a single SD slot so no chance of mixing it up with another)

    I knew i had forgotten something..

    from the pis /etc/fstab


    proc /proc proc defaults 0 0 PARTUUID=b8c9fbb7-01 /boot vfat defaults 0 2 PARTUUID=b8c9fbb7-02 / ext4 defaults,noatime 0 1

    'blkid' will show your partition devices with their UUIDs.

    Yeah. Easy enough to either label a new card with the old ids or modify
    fstab.

    BTDTGTTS

    What about /dev? Is that dynamically created or not?

    A dynamic filesystem mounted to a static pre-existing mount point.

    OK, thanks

    Only question left unanswered is whether there is anything else on the
    SD cardstructure to identify which partition is the boot one?

    Does the boot loader simply look for the only (first?) VFAT partition?

    Theo

    --
    “Some people like to travel by train because it combines the slowness of
    a car with the cramped public exposure of 
an airplane.”

    Dennis Miller

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to Computer Nerd Kev on Wed Sep 20 04:52:43 2023
    On 19/09/2023 23:19, Computer Nerd Kev wrote:
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    On 19/09/2023 10:11, Computer Nerd Kev wrote:
    The Natural Philosopher <tnp@invalid.invalid> wrote:
    The Pi doesn't need anything like a boot sector. The firmware is
    smart enough to find the boot files on the filesystem, provided
    it's FAT32.

    I suppose that needs to be the first partition?

    "It is no longer necessary for the first partition to be the FAT
    partition, as the MSD boot will continue to search for a FAT
    partition beyond the first one."
    https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#boot-sequence

    Ah. The Final Question is answered.

    Thanks m8

    The other question why always gets me on backups is how many of the
    directories in the root partition can safely be ignored?

    /sys, /proc, /run would seem to be created by the machine at boot time,
    and shouldn't be created as subdirs of the root partition...or should
    the directories be created, but empty?

    /sys and /proc can be created empty, because they're mountpoints.
    However I don't bother excluding them from backups personally.

    So putatively one might back up everything in / except /run, /proc &
    /sys, including /boot, plonk a shiny new SD card in a host computer,
    create a VFAT partition of say 256MB, and the rest as ext4, untar the
    archive into the ext4 partition and move the entire contents of /boot
    onto the VFAT, shove the card in a pi , apply power, and Robert should
    be a relative?

    The vfat partition will get mounted on the (now empty) /boot mount
    point, and everything else will be where it ought to be, and /run /sys
    and /proc will get generated by the boot process?

    Yes. Mounting the FAT32 partition at /boot is specific behaviour
    for RPi OS / Debian. Linux itself doesn't care if that partition
    gets mounted at all - by the time the kernel is loaded its job has
    been done, but Debian probably expects it there for configuration
    and upgrades.

    What about /dev? Is that dynamically created or not?

    The device files need backing up, or else something might have to
    create them with mknod. It contains more mountpoints though, so
    you can look at the list displayed by "mount" and exclude those sub-directories of /dev if you like. Again, I don't bother to do
    that.

    One presumes that if anything else is recreated in /dev at boot, it will
    wipe out or be mounted over what was 'backed up'.

    Well the strategy seems clear: At some point I will set up a 'preserve
    this SD card' methodology, buy a new SD card and see if I can make it work.



    --
    “A leader is best When people barely know he exists. Of a good leader,
    who talks little,When his work is done, his aim fulfilled,They will say,
    “We did this ourselves.”

    ― Lao Tzu, Tao Te Ching

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Hermann Riemann@3:770/3 to All on Wed Sep 20 09:45:21 2023
    Am 17.09.23 um 11:19 schrieb The Natural Philosopher:

    It occurs to me that essentially a copy of the contents of the two PI partitions would be enough, together with an install of Raspios,
    followed by a copy back to the fresh SD, to clone a setup.

    It seems too easy. What am I missing?


    Using an other Linux computer
    ( raspberry pi 400 or PC ),
    take an 5 TB hard disk,
    an use dd in order to copy the SD card.
    This may be enough for 300 different copy 16 GB SD card.

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Charlie Gibbs@3:770/3 to Newyana2@invalid.nospam on Wed Sep 20 17:23:36 2023
    On 2023-09-19, Newyana2 <Newyana2@invalid.nospam> wrote:

    "The Natural Philosopher" <tnp@invalid.invalid> wrote

    Start menu. The raspberry icon in lower left.

    ??? Icon? On a headless Pi Zero?

    Ah. You might have said that in the first place.
    It doesn't make much sense to ask for help and then
    be difficult with people who try to help. I gather
    you only posted to argue.

    This is totally off topic, but I'm curious: did you
    once post under the name "Mayayana"? If so, I'd like
    to thank you for the quote in my .sig below.

    --
    /~\ Charlie Gibbs | They don't understand Microsoft
    \ / <cgibbs@kltpzyxm.invalid> | has stolen their car and parked
    X I'm really at ac.dekanfrus | a taxi in their driveway.
    / \ if you read it the right way. | -- Mayayana

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to Charlie Gibbs on Thu Sep 21 08:38:19 2023
    On 20/09/2023 18:23, Charlie Gibbs wrote:
    On 2023-09-19, Newyana2 <Newyana2@invalid.nospam> wrote:

    "The Natural Philosopher" <tnp@invalid.invalid> wrote

    Start menu. The raspberry icon in lower left.

    ??? Icon? On a headless Pi Zero?

    Ah. You might have said that in the first place.
    It doesn't make much sense to ask for help and then
    be difficult with people who try to help. I gather
    you only posted to argue.

    This is totally off topic, but I'm curious: did you
    once post under the name "Mayayana"? If so, I'd like
    to thank you for the quote in my .sig below.


    Er no. I don't believe I ever did. Ive been using this one a LONG time
    now as my 'I'm not at work presenting my company' one for a long time.


    --
    "Women actually are capable of being far more than the feminists will
    let them."

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Anssi Saari@3:770/3 to The Natural Philosopher on Thu Sep 21 22:16:29 2023
    The Natural Philosopher <tnp@invalid.invalid> writes:

    The other question why always gets me on backups is how many of the directories in the root partition can safely be ignored?

    If you don't want to fiddle with that, there is partclone which makes
    copies of partitions, smartly enough so it doesn't copy unused sectors.

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From Jim Jackson@3:770/3 to druck on Thu Sep 21 20:58:01 2023
    On 2023-09-19, druck <news@druck.org.uk> wrote:

    Here's the rsync options I used to backup the partitions, firstly the
    FAT partition which accounts for the lower resolution of timestamps

    rsync -vax --stats --modify-window=2 --delete /boot/ <boot_backup_dir>

    And the main partition with the exclusions to prevent pseudo filing
    systems and temporary stuff being included when backing up live systems,
    some of it shouldn't be necessary but turns out is needed

    rsync -vaxHAX --stats --numeric-ids --delete --delete-excluded --exclude=/tmp/*
    --exclude=/run/*
    --include=/var/lock
    --exclude=/var/run
    --exclude=/var/swap
    --exclude=/var/cache/*
    --exclude=/var/tmp/*
    --exclude=/var/lib/machines/*/var/cache/
    --exclude=/var/lib/mlocate/*
    --exclude=/var/lib/nginx/fastcgi/*
    --exclude=/var/log/journal/*
    --exclude=/var/log/Xorg.*
    --exclude=/lost+found/*
    --exclude=*/.cache
    --exclude=*/__pycache__
    --exclude=*/chromium/hyphen-data/
    --exclude=*/.config/chromium/*Cache*/*
    / <root_backup_dir>


    I use very similar options. I add -u and I add the --delete-after
    option. Oh and --numeric-ids . I've been bitten by backing up to a
    system where some of the system uids (<1000) were different and on backup
    got mangled. At least with numeric-only they go back as they were.

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)
  • From The Natural Philosopher@3:770/3 to Jim Jackson on Fri Sep 22 09:40:59 2023
    On 21/09/2023 21:58, Jim Jackson wrote:
    On 2023-09-19, druck <news@druck.org.uk> wrote:

    Here's the rsync options I used to backup the partitions, firstly the
    FAT partition which accounts for the lower resolution of timestamps

    rsync -vax --stats --modify-window=2 --delete /boot/ <boot_backup_dir>

    And the main partition with the exclusions to prevent pseudo filing
    systems and temporary stuff being included when backing up live systems,
    some of it shouldn't be necessary but turns out is needed

    rsync -vaxHAX --stats --numeric-ids --delete --delete-excluded
    --exclude=/tmp/*
    --exclude=/run/*
    --include=/var/lock
    --exclude=/var/run
    --exclude=/var/swap
    --exclude=/var/cache/*
    --exclude=/var/tmp/*
    --exclude=/var/lib/machines/*/var/cache/
    --exclude=/var/lib/mlocate/*
    --exclude=/var/lib/nginx/fastcgi/*
    --exclude=/var/log/journal/*
    --exclude=/var/log/Xorg.*
    --exclude=/lost+found/*
    --exclude=*/.cache
    --exclude=*/__pycache__
    --exclude=*/chromium/hyphen-data/
    --exclude=*/.config/chromium/*Cache*/*
    / <root_backup_dir>


    I use very similar options. I add -u and I add the --delete-after
    option. Oh and --numeric-ids . I've been bitten by backing up to a
    system where some of the system uids (<1000) were different and on backup
    got mangled. At least with numeric-only they go back as they were.



    This is what I ended up using. I have to back up /boot every time as it
    gets deleted by the first backup

    #!/bin/bash
    # Mounts the backup server, backs up the files in / excluding mounted
    stuff and other dross
    # then backs up the boot partition
    # and unmounts the backup server


    mount 192.168.0.100:/backup2 /mnt
    cd /mnt/heating-controller

    # And the main partition with the exclusions to prevent pseudo filing
    systems and temporary stuff being included when backing up live systems,
    some of it shouldn't be necessary # but turns out is needed

    rsync -vaxHAX --stats --numeric-ids --delete --delete-excluded --one-file-system --exclude=/tmp/* --exclude=/run/* --include=/var/lock --exclude=/var/run --exclude=/var/swap --exclude=/var/cache/* --exclude=/var/tmp/* --exclude=/var/lib/machines/*/var/cache/ --exclude=/var/lib/mlocate/* --exclude=/var/lib/nginx/fastcgi/* --exclude=/var/log/journal/* --exclude=/var/log/Xorg.*
    --exclude=/lost+found/* --exclude=*/.cache --exclude=*/__pycache__ --exclude=*/chromium/hyphen-data/ --exclude=*/.config/chromium/*Cache*/* / .

    rsync -vax --stats --modify-window=2 --delete /boot/ ./boot
    cd
    umount /mnt

    --
    “I know that most men, including those at ease with problems of the greatest complexity, can seldom accept even the simplest and most
    obvious truth if it be such as would oblige them to admit the falsity of conclusions which they have delighted in explaining to colleagues, which
    they have proudly taught to others, and which they have woven, thread by thread, into the fabric of their lives.”

    ― Leo Tolstoy

    --- SoupGate-Win32 v1.05
    * Origin: Agency HUB, Dunedin - New Zealand | Fido<>Usenet Gateway (3:770/3)