New GNUnet publication program

[Kenneth Almquist]

I've written a variant of gnunet-insert, which I've named gnunet-pub.
The key idea is to split the publishing operation into two steps:
1)  Determine precisely what to publish, and
2)  Actually publish it.

A given call to gnunet-pub can perform one or both of these steps.
If you say

    gnunet-pub -o spec file...

gnunet-pub will write a publication specification to the file spec.
You can then say

    gnunet-pub -i spec -P

where the -P option tells gnunet-pub to actually publish the files
to GNUnet.  The command

    gnunet-pub -P file...

will perform both operations in one step.

This has a number of benefits.

The first is transparency.  The user can create a specification file
and then see exactly what will be published, before performing the
publication.  I feel that this is quite important because publishing
is an irreversible operation.

The second benefit is that the front end (the part of gnunet-pub
which generates the specification) can easily be replaced.  There
is no need to use a programming language that can interface to C;
any programming language that is capable of writing a text file
can be used.  And testing is straightforward, because all you have
to do is look at the text file generated by the front end to see
if it is correct.

A third benefit is that, if the user retains the specification file
and the published data files, the published content can be recreated.
This makes it simple to re-publish content if the GNUnet server's
data base ever crashes.

The ability to recreate published content has other possible uses.
Currently, you can unindex a file, but the IBlocks and search records
pointing to the file will still be in your data base.  With a
spacification file, it would be possible to delete all of the
published blocks using the DELETE message in the FS client-server
protocol.

Conversely, you might want to be sure that the blocks you have
published are still in the server data base, and haven't been pushed
out by other content, or deleted due to software bugs or hardware
glitches.  If the FS client-server protocol were enhanced to include
a query to determine whether a block was in the data base, you could
periodically check whether all of the data you published was still in
the data base.


To get the code, documentation, and test cases:

1)  Go to a directory containing the GNUnet source, then cd to
    src/applications/fs.

2)  Download http://mysite.verizon.net/vzeefhzu/gnunet-pub.tar.bz2

3)  Untar the file.  This will create src/applications/fs/gnunet-pub.

There are probably a number of issues with this code due to the lack
of a complete design before coding.  On the other hand, I think that
the set of test cases is large enough to ensure basic usability.  All
feedback welcome.

 - Kenneth Almqist


_________________________________________________________________
Windows Live: Keep your friends up to date with what you do online.
http://windowslive.com/Campaign/SocialNetworking?ocid=PID23285::T:WLMTAGL:ON:WL:en-US:SI_SB_online:082009

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@...
http://lists.gnu.org/mailman/listinfo/gnunet-developers

[Christian Grothoff]

Hi!

This is interesting, in part because I recently did a similar split into two
steps for the new 0.9.x FS publishing API.  Now, what is not yet possible with
that API is the serialization into a specification file for the end-user between
the two stages, but that might be a possibility. The main problem I see is
that the new API allows for "files" to be published that are not on disk (the
caller would just have to specify a function for how to get the data); that
kind-of conflicts with the ability to serialize to a human-readable file.  But
by serializing to a human-readable file + archive with embedded files this can
be supported sufficiently well.   So the idea is clearly worthwhile, I just
think it should not be done with hand-hacked clients but within the FS
library.

Also, I think (/hope) you're wrong about IBlocks not being deleted by gnunet-
unindex.  That'd certainly not be by design.  Finally, I wonder if the ability
to delete KBlocks is really all that important since I suspect users won't
keep the specification files around.  But maybe I'm wrong about that, and it is
certainly a reasonable idea, especially if we integrate it with a way to
trigger unindexing from tools like gnunet-auto-share (where code may control
the generation and management of those specs).


My 2 cents

Christian

On Tuesday 01 September 2009 09:33:00 Kenneth Almquist wrote:

> I've written a variant of gnunet-insert, which I've named gnunet-pub.
> The key idea is to split the publishing operation into two steps:
> 1)  Determine precisely what to publish, and
> 2)  Actually publish it.
>
> A given call to gnunet-pub can perform one or both of these steps.
> If you say
>
>     gnunet-pub -o spec file...
>
> gnunet-pub will write a publication specification to the file spec.
> You can then say
>
>     gnunet-pub -i spec -P
>
> where the -P option tells gnunet-pub to actually publish the files
> to GNUnet.  The command
>
>     gnunet-pub -P file...
>
> will perform both operations in one step.
>
> This has a number of benefits.
>
> The first is transparency.  The user can create a specification file
> and then see exactly what will be published, before performing the
> publication.  I feel that this is quite important because publishing
> is an irreversible operation.
>
> The second benefit is that the front end (the part of gnunet-pub
> which generates the specification) can easily be replaced.  There
> is no need to use a programming language that can interface to C;
> any programming language that is capable of writing a text file
> can be used.  And testing is straightforward, because all you have
> to do is look at the text file generated by the front end to see
> if it is correct.
>
> A third benefit is that, if the user retains the specification file
> and the published data files, the published content can be recreated.
> This makes it simple to re-publish content if the GNUnet server's
> data base ever crashes.
>
> The ability to recreate published content has other possible uses.
> Currently, you can unindex a file, but the IBlocks and search records
> pointing to the file will still be in your data base.  With a
> spacification file, it would be possible to delete all of the
> published blocks using the DELETE message in the FS client-server
> protocol.
>
> Conversely, you might want to be sure that the blocks you have
> published are still in the server data base, and haven't been pushed
> out by other content, or deleted due to software bugs or hardware
> glitches.  If the FS client-server protocol were enhanced to include
> a query to determine whether a block was in the data base, you could
> periodically check whether all of the data you published was still in
> the data base.
>
>
> To get the code, documentation, and test cases:
>
> 1)  Go to a directory containing the GNUnet source, then cd to
>     src/applications/fs.
>
> 2)  Download http://mysite.verizon.net/vzeefhzu/gnunet-pub.tar.bz2
>
> 3)  Untar the file.  This will create src/applications/fs/gnunet-pub.
>
> There are probably a number of issues with this code due to the lack
> of a complete design before coding.  On the other hand, I think that
> the set of test cases is large enough to ensure basic usability.  All
> feedback welcome.
>
>  - Kenneth Almqist
>
>
> _________________________________________________________________
> Windows Live: Keep your friends up to date with what you do online.
> http://windowslive.com/Campaign/SocialNetworking?ocid=PID23285::T:WLMTAGL:O
> N:WL:en-US:SI_SB_online:082009
>
> _______________________________________________
> GNUnet-developers mailing list
> GNUnet-developers@...
> http://lists.gnu.org/mailman/listinfo/gnunet-developers
>

--
http://grothoff.org/christian/


_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@...
http://lists.gnu.org/mailman/listinfo/gnunet-developers

[Kenneth Almquist]

> This is interesting, in part because I recently did a similar split into two
> steps for the new 0.9.x FS publishing API. Now, what is not yet possible with
> that API is the serialization into a specification file for the end-user between
> the two stages, but that might be a possibility. The main problem I see is
> that the new API allows for "files" to be published that are not on disk (the
> caller would just have to specify a function for how to get the data); that
> kind-of conflicts with the ability to serialize to a human-readable file.

I haven't seen anything about the new API on the web site; do you
have a draft written yet?

The way I handle files that are not on disk is to write them to files
at the same time the specification file is written.  Basicly, if you give
a specification file name "a/b/c", then the specification is written
to that file, and any data that isn't already in files is written to files
named "a/b/0001.dat", "a/b/0002.gnd", and so on.  In order to make
this a bit more convenient to put each specification file in its own
directory, the directory "a/b" is created if it does not exist.  If higher
level directories (such as the directory "a" in "a/b/c") do not exist,
that is treated as an error.

The other issue with files is that the current working directory of a
process which reads a specification file might be different than the
current working directory of the program that wrote the file.  So
relative file names in a specification file are relative to the directory
containing the specification file.


_________________________________________________________________
Hotmail: Free, trusted and rich email service.
http://clk.atdmt.com/GBL/go/171222984/direct/01/

_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@...
http://lists.gnu.org/mailman/listinfo/gnunet-developers

[Christian Grothoff-4]

On Monday 14 September 2009 17:38:46 you wrote:

> > This is interesting, in part because I recently did a similar split into
> > two steps for the new 0.9.x FS publishing API. Now, what is not yet
> > possible with that API is the serialization into a specification file for
> > the end-user between the two stages, but that might be a possibility. The
> > main problem I see is that the new API allows for "files" to be published
> > that are not on disk (the caller would just have to specify a function
> > for how to get the data); that kind-of conflicts with the ability to
> > serialize to a human-readable file.
>
> I haven't seen anything about the new API on the web site; do you
> have a draft written yet?

Yes: https://gnunet.org/svn/gnunet/src/include/gnunet_fs_service.h
 
> The way I handle files that are not on disk is to write them to files
> at the same time the specification file is written.  Basicly, if you give
> a specification file name "a/b/c", then the specification is written
> to that file, and any data that isn't already in files is written to files
> named "a/b/0001.dat", "a/b/0002.gnd", and so on.  In order to make
> this a bit more convenient to put each specification file in its own
> directory, the directory "a/b" is created if it does not exist.  If higher
> level directories (such as the directory "a" in "a/b/c") do not exist,
> that is treated as an error.

Right, one should write them to disk.  Now, I'm not sure about why you need to
generate errors (never good) if higher-leveld dirs don't exist; since this
would be all relative to the location of the specification file, we could just
create those dirs.

> The other issue with files is that the current working directory of a
> process which reads a specification file might be different than the
> current working directory of the program that wrote the file.  So
> relative file names in a specification file are relative to the directory
> containing the specification file.

I agree, good thing to remember when writing that code...

Best,

Christian
--
http://grothoff.org/christian/


_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@...
http://lists.gnu.org/mailman/listinfo/gnunet-developers

[Christian Grothoff]

On Monday 14 September 2009 17:38:46 you wrote:

> > This is interesting, in part because I recently did a similar split into
> > two steps for the new 0.9.x FS publishing API. Now, what is not yet
> > possible with that API is the serialization into a specification file for
> > the end-user between the two stages, but that might be a possibility. The
> > main problem I see is that the new API allows for "files" to be published
> > that are not on disk (the caller would just have to specify a function
> > for how to get the data); that kind-of conflicts with the ability to
> > serialize to a human-readable file.
>
> I haven't seen anything about the new API on the web site; do you
> have a draft written yet?

Yes: https://gnunet.org/svn/gnunet/src/include/gnunet_fs_service.h
 
> The way I handle files that are not on disk is to write them to files
> at the same time the specification file is written.  Basicly, if you give
> a specification file name "a/b/c", then the specification is written
> to that file, and any data that isn't already in files is written to files
> named "a/b/0001.dat", "a/b/0002.gnd", and so on.  In order to make
> this a bit more convenient to put each specification file in its own
> directory, the directory "a/b" is created if it does not exist.  If higher
> level directories (such as the directory "a" in "a/b/c") do not exist,
> that is treated as an error.

Right, one should write them to disk.  Now, I'm not sure about why you need to
generate errors (never good) if higher-leveld dirs don't exist; since this
would be all relative to the location of the specification file, we could just
create those dirs.

> The other issue with files is that the current working directory of a
> process which reads a specification file might be different than the
> current working directory of the program that wrote the file.  So
> relative file names in a specification file are relative to the directory
> containing the specification file.

I agree, good thing to remember when writing that code...

Best,

Christian
--
http://grothoff.org/christian/


_______________________________________________
GNUnet-developers mailing list
GNUnet-developers@...
http://lists.gnu.org/mailman/listinfo/gnunet-developers