es-srme/es-srme.8��0100644�0001750�0001750�00000022124�14221655151�0012617�0��ustar�00john��john��.Dd $Mdocdate: April 1 2022 $

.Dt ES-SRME 8

.Os

.Sh NAME

.Nm es-srme

.Nd display and modify metadata of softraid volumes

.Sh SYNOPSIS

.Nm

.Op Ar file

.Op Ar new size

.Sh DESCRIPTION

The main use of

.Nm

is to modify the size fields in the metadata of a softraid volume, after the RAID partition containing it has been resized with /sbin/disklabel. The process is somewhat analogous to using /sbin/growfs to enlarge an FFS filesystem after extending it's partition in the same way.

.Pp

The

.Nm

utility can also be used to inspect the volume checksum and chunk checksum of a softraid volume, and to detect and correct a specific type of invalid chunk checksum data that has been written by a kernel version which has a known bug.

.Pp

The

.Ar file

argument should be a regular file of exactly 512 bytes, extracted from the first metadata block of a softraid volume.

This will typically be done using a command such as:

.Pp

.Dl # dd if=/dev/rsd0d of=metadata skip=16 count=1

.Pp

Where

.Ar rsd0d

is a RAID partition.

.Nm

can then be used to inspect the volume sizes stored in the metadata, and validate the checksums:

.Pp

.Dl # es-srme metadata

.Dl "Read magic: 4d4152436372616d"

.Dl "Volume size is 1953124472 sectors, 976562236K, 953674M, 931G"

.Dl "Chunk size is 1953124472 sectors, 976562236K, 953674M, 931G"

.Dl "Coerced size is 1953124472 sectors, 976562236K, 953674M, 931G"

.Dl "Calculated volume MD5: 5f819d7cd369af9d/d693b4e311791081"

.Dl " Stored volume MD5: 5f819d7cd369af9d/d693b4e311791081"

.Dl "Calculated chunk MD5: 459d3f38167c4119/a3cae6705a4e0a97"

.Dl " Stored chunk MD5: 459d3f38167c4119/a3cae6705a4e0a97"

.Pp

A bug in certain versions of the OpenBSD kernel causes an invalid checksum to be calculated and stored for the chunk metadata.

.Nm

can detect this, and will fix it by writing the correct checksum to the corresponding metadata field when the volume size is changed by the user.

.Pp

Modifying the size fields of the metadata will usually be done after first resizing the RAID partition that contains the softraid volume using /sbin/disklabel.

The required size is typically 528 blocks less than the size of the containing volume. For example, considering the following RAID partition:

.Pp

.Dl "# size offset fstype [fsize bsize cpg]"

.Dl " c: 7812500000 0 unused"

.Dl " d: 7812500000 0 RAID"

.Pp

The required size for the softraid partition would likely be 7812500000-528=7812499472.

Invoking

.Nm

with this figure for the second argument will produce output similar to the following:

.Pp

.Dl # es-srme metadata 7812499472

.Dl "Read magic: 4d4152436372616d"

.Dl "Volume size is 1953124472 sectors, 976562236K, 953674M, 931G"

.Dl "Chunk size is 1953124472 sectors, 976562236K, 953674M, 931G"

.Dl "Coerced size is 1953124472 sectors, 976562236K, 953674M, 931G"

.Dl "Calculated volume MD5: 5f819d7cd369af9d/d693b4e311791081"

.Dl " Stored volume MD5: 5f819d7cd369af9d/d693b4e311791081"

.Dl "Calculated chunk MD5: 459d3f38167c4119/a3cae6705a4e0a97"

.Dl " Stored chunk MD5: 459d3f38167c4119/a3cae6705a4e0a97"

.Dl "New size is 7812499472 sectors, 3906249736K, 3814697M, 3725G"

.Dl " New volume MD5: 61f26ecb2f4684f2/fe1c3473a897fb5d"

.Dl " New chunk MD5: 94a384417b8e3b10/ecaa2ed5cde078f8"

.Pp

At this point, the updated metadata in the file can be written back to the same block it was read from on the RAID partition:

.Pp

.Dl # dd if=metadata of=/dev/rsd0d seek=16 count=1

.Pp

This should be performed with the softraid volume detached.

The softraid volume can then be re-attached, and should report it's new size on the console:

.Pp

.Dl # bioctl -c C -l sd0d softraid0

.Dl Passphrase:

.Dl softraid0: CRYPTO volume attached as sd5

.Pp

.Dl sd5 at scsibus3 targ 3 lun 0: <OPENBSD, SR CRYPTO, 006>

.Dl sd5: 3814697MB, 512 bytes/sector, 7812499472 sectors

.Sh EXIT STATUS

.Dl 0 - Success, no data written.

.Dl 1 - Success, no data written, buggy chunk metadata checksum detected.

.Dl 2 - Success, size fields updated.

.Dl 3 - Failure, an error occurred.

.Sh DIAGNOSTICS

.Bl -diag

.It "Error allocating buffers"

Memory allocation failed.

.It "Error opening file for reading"

The supplied

.Ar file

could not be opened for reading.

.It "Error reading 512 bytes from file"

Reading the supplied

.Ar file

failed, or returned fewer than 512 bytes of data.

.It "Bad magic, should be 0x4d4152436372616d"

The first eight bytes of the supplied file don't match the expected magic. A likely cause is that the wrong sector has been read from the RAID partition.

.It "Metadata version is X, only version 6 is supported by this program"

Currently,

.Nm

only supports version 6 softraid metadata.

.It "This doesn't appear to be a softraid crypto volume"

Currently,

.Nm

only supports softraid crypto volumes. Other disciplines such as RAID-1, and RAID-5, are not yet supported.

.It "Mismatch between size fields, exiting..."

The supplied

.Ar file

appears to contain valid softraid metadata for a softraid crypto volume, but the values of the three size fields are not all identical as would be expected from such a volume.

.It "New size argument is invalid"

The

.Ar new size

argument is either non-numeric, zero, negative, or exceeds 2^62.

.It "Error opening file for writing"

The supplied

.Ar file

could not be opened for writing.

.It "Error writing 512 bytes to file"

Writing the modified data back to

.Ar file

failed, or fewer than 512 bytes of data were written.

.It "The chunk checksum matches bytes 168 - 184, instead of bytes 168 - 240"

.Nm

has detected that the volume checksum has been calculated from the wrong byterange. This is an informational message, and does not stop execution of the program. If

.Nm

was invoked with a

.Ar new size

parameter, then correct checksum data will be written along with it.

.It "Note: supplied size value is equal to existing value!"

The supplied value for

.Ar new size

is the same as the existing size. This is an informational message, and does not stop execution of the program. The metadata will still be re-written, and invoking

.Nm

in this way can be useful in order to overwrite an invalid volume checksum as described above with a valid one.

.El

.Sh SEE ALSO

.Rs

.%Q Exotic Silicon

.%T Resizing softraid volumes

.%O Supporting material, and official webpage, (material also available via gemini)

.%A Crystal Kolipe

.%U https://www.exoticsilicon.com/research/resizing_softraid_volumes

.%U gemini://gemini.exoticsilicon.com/research/resizing_softraid_volumes

.%D 2022

.Re

.Sh HISTORY

The initial version of

.Nm

was written in March 2022.

.Sh AUTHORS

.An Crystal Kolipe

.Aq kolipe.c@exoticsilicon.com

.Sh CAVEATS

.Nm

intentionally does not stop processing if an invalid checksum is read from

.Ar file

and no specific warning is displayed, (except in the case of the chunk checksum matching the incorrect byte-range described above). The only indication that the checksum is wrong will be a mis-match between the values displayed for

.Sy calculated MD5

and

.Sy stored MD5 .

.Sh LICENSE

.Nm

and this manual page are distributed under the following license:

.Pp

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

.Pp

This software is licensed exclusively under this specific license text. The license text may not be changed, and the software including modified versions may not be re-licensed under any other license text.

.Pp

Redistributions of source code must retain the above copyright notice, this list of conditions, and the following disclaimer.

.Pp

Redistributions in binary form must reproduce the above copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution.

.Pp

All advertising materials mentioning features or use of this software must display the following acknowledgement: This product includes software developed by Exotic Silicon.

.Pp

The name of Exotic Silicon must not be used to endorse or promote products derived from this software without specific prior written permission.

.Pp

Redistributions of modified versions of the source code must be clearly identified as having been modified from the original.

.Pp

Redistributions in binary form that have been created from modified versions of the source code must clearly state in the documentation and/or other materials provided with the distribution that the source code has been modified from the original.

.Pp

THIS SOFTWARE IS PROVIDED 'AS IS' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL EXOTIC SILICON BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

��es-srme/es-srme.c��0100644�0001750�0001750�00000020212�14221650245�0012665�0��ustar�00john��john��/*

Redistribution and use in source and binary forms, with or without modification,

are permitted provided that the following conditions are met:

This software is licensed exclusively under this specific license text. The

license text may not be changed, and the software including modified versions

may not be re-licensed under any other license text.

Redistributions of source code must retain the above copyright notice, this

list of conditions, and the following disclaimer.

Redistributions in binary form must reproduce the above copyright notice,

this list of conditions, and the following disclaimer in the documentation

and/or other materials provided with the distribution.

All advertising materials mentioning features or use of this software must

display the following acknowledgement: This product includes software

developed by Exotic Silicon.

The name of Exotic Silicon must not be used to endorse or promote products

derived from this software without specific prior written permission.

Redistributions of modified versions of the source code must be clearly

identified as having been modified from the original.

Redistributions in binary form that have been created from modified versions

of the source code must clearly state in the documentation and/or other

materials provided with the distribution that the source code has been

modified from the original.

THIS SOFTWARE IS PROVIDED 'AS IS' AND ANY EXPRESS OR IMPLIED WARRANTIES,

INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND

FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL

EXOTIC SILICON BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,

EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT

OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA OR PROFITS; OR BUSINESS

INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN

CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING

IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY

OF SUCH DAMAGE.

/* This is the source code for es-srme version 1.0. */

/* For more details please visit: */

/* https://www.exoticsilicon.com/research/resizing_softraid_volumes */

/* or gemini://gemini.exoticsilicon.com/research/resizing_softraid_volumes */

include <stdio.h>

include <fcntl.h>

include <unistd.h>

include <stdlib.h>

include <sys/types.h>

include <md5.h>

/* Magic number to identify softraid metadata */

define MAGIC (uint64_t)0x4d4152436372616d

void usage(char * progname)

{

printf ("Usage: %s [input] [newsize]\n", progname);

printf ("Where [input] is a 512 byte softraid partition header, usually found in\nsector 16 of a RAID partition.\n");

printf ("If [newsize] is specified, new size values are written back to the header,\notherwise the current size values are simply displayed.\n");

return ;

}

int main(int argc, char *argv[], char *env[])

{

int fd;

unsigned char * buffer;

unsigned char * buffer_md5;

unsigned int raid_level;

unsigned int version;

unsigned int flag_bad_checksum;

uint64_t magic;

uint64_t size_vol;

uint64_t size_chunk;

uint64_t size_coerced;

uint64_t size_new;

MD5_CTX cont;

if (argc<2 || argc>3) {

usage(argv[0]);

return (3);

flag_bad_checksum=0;

buffer=malloc(512);

buffer_md5=malloc(MD5_DIGEST_LENGTH);

if (buffer == NULL || buffer_md5 == NULL) {

dprintf (STDERR_FILENO, "Error allocating buffers.\n");

return (3);

fd = open (argv[1], O_RDONLY);

if (fd == -1) {

dprintf (STDERR_FILENO, "Error opening %s for reading.\n", argv[1]);

return (3);

if (read (fd, buffer, 512) != 512) {

dprintf (STDERR_FILENO, "Error reading 512 bytes from %s.\n", argv[1]);

return (3);

close (fd);

/* Check magic */

magic=*(uint64_t *)buffer;

printf ("Read magic: %llx\n", magic);

if (magic != MAGIC) {

dprintf (STDERR_FILENO, "Bad magic, should be %llx.\n", MAGIC);

return (3);

/* We only support version 6 metadata */

version=*(long *)(buffer+8);

if (version != 0x06) {

dprintf (STDERR_FILENO, "Metadata version is %x, only version 6 is supported by this program.\n", version);

return (3);

/* We only support the crypto discipline */

raid_level=*(long *)(buffer+52);

if (raid_level != 0x43) {

dprintf (STDERR_FILENO, "This doesn't appear to be a softraid crypto volume.\n");

return (3);

size_vol=*(uint64_t *)(buffer+56);

size_chunk=*(uint64_t *)(buffer+208);

size_coerced=*(uint64_t *)(buffer+216);

printf ("Volume size is %llu sectors, %lluK, %lluM, %lluG\n",size_vol, size_vol/2, size_vol/2048, size_vol/2097152);

printf ("Chunk size is %llu sectors, %lluK, %lluM, %lluG\n",size_chunk, size_chunk/2, size_chunk/2048, size_chunk/2097152);

printf ("Coerced size is %llu sectors, %lluK, %lluM, %lluG\n",size_coerced, size_coerced/2, size_coerced/2048, size_coerced/2097152);

if (!((size_vol==size_chunk) && (size_chunk==size_coerced))) {

dprintf (STDERR_FILENO, "Mismatch between size fields, exiting...\n");

return (3);

/* Calculate the MD5 checksum of the first 96 bytes, which is the invariant volume metadata */

MD5Init (&cont);

MD5Update (&cont, buffer, 96);

MD5Final (buffer_md5, &cont);

printf ("Calculated volume MD5: %llx/%llx\n",*(uint64_t )buffer_md5,(uint64_t *)(buffer_md5+8));

printf (" Stored volume MD5: %llx/%llx\n",*(uint64_t )(buffer+96),(uint64_t *)(buffer+104));

/* Calculate the MD5 checksum of bytes 168 - 184 */

/* Certain buggy kernels calculate and store the wrong checksum in the chunk MD5 field */

/* We take the opportunity to detect this and optionally fix it */

MD5Init (&cont);

MD5Update (&cont, (buffer+168), 16);

MD5Final (buffer_md5, &cont);

if ( (*(uint64_t )buffer_md5==(uint64_t )(buffer+240)) && ((uint64_t )(buffer_md5+8)==(uint64_t *)(buffer+248)) ) {

printf ("Warning: This metadata seems to have been created with a buggy OpenBSD kernel!\n");

printf ("The chunk checksum matches bytes 168 - 184, instead of bytes 168 - 240.\n");

flag_bad_checksum=1;

/* Calculate the MD5 checksum of bytes 168 - 240, which is the invariant chunk metadata */

MD5Init (&cont);

MD5Update (&cont, (buffer+168), 72);

MD5Final (buffer_md5, &cont);

printf ("Calculated chunk MD5: %llx/%llx\n",*(uint64_t )buffer_md5,(uint64_t *)(buffer_md5+8));

printf (" Stored chunk MD5: %llx/%llx\n",*(uint64_t )(buffer+240),(uint64_t *)(buffer+248));

/* Exit here if we are in read-only mode, I.E. no new size was specified. */

if (argc==2) {

return (flag_bad_checksum);

/* Parse new size argument */

size_new=strtonum(argv[2],0,(uint64_t)1<<62,NULL);

if (size_new == 0) {

dprintf (STDERR_FILENO, "New size argument is invalid.\n");

return (3);

if (size_new == size_vol) {

printf ("Note: supplied size value is equal to existing value!\n");

/* Write new data to size fields */

printf ("New size is %llu sectors, %lluK, %lluM, %lluG\n",size_new, size_new/2, size_new/2048, size_new/2097152);

(uint64_t *)(buffer+56)=size_new;
(uint64_t *)(buffer+208)=size_new;
(uint64_t *)(buffer+216)=size_new;

/* Re-calculate both checksums, and write them to the in-memory buffer */

/* Re-calculate the MD5 checksum of the first 96 bytes, which is the invariant volume metadata */

MD5Init (&cont);

MD5Update (&cont, buffer, 96);

MD5Final (buffer_md5, &cont);

printf (" New volume MD5: %llx/%llx\n",*(uint64_t )buffer_md5,(uint64_t *)(buffer_md5+8));

(uint64_t )(buffer+96)=(uint64_t *)buffer_md5;
(uint64_t )(buffer+104)=(uint64_t *)(buffer_md5+8);

/* Calculate the MD5 checksum of bytes 168 - 240, which is the invariant chunk metadata */

MD5Init (&cont);

MD5Update (&cont, (buffer+168), 72);

MD5Final (buffer_md5, &cont);

printf (" New chunk MD5: %llx/%llx\n",*(uint64_t )buffer_md5,(uint64_t *)(buffer_md5+8));

(uint64_t )(buffer+240)=(uint64_t *)buffer_md5;
(uint64_t )(buffer+248)=(uint64_t *)(buffer_md5+8);

/* Write the in-memory buffer over the original input file */

fd = open (argv[1], O_WRONLY | O_TRUNC);

if (fd == -1) {

dprintf (STDERR_FILENO, "Error opening %s for writing.\n", argv[1]);

return (3);

if (write (fd, buffer, 512) != 512) {

dprintf (STDERR_FILENO, "Error writing 512 bytes to %s.\n", argv[1]);

return (3);

close (fd);

return (2);

}

��

Proxy Information

Original URL: gemini://gemini.exoticsilicon.com/downloads/es-srme_v1.0.tar
Status Code: Success (20)
Meta: application/octet-stream
Capsule Response Time: 342.628147 milliseconds
Gemini-to-HTML Time: 8.175902 milliseconds

This content has been proxied by September (3851b).