[NSE] Script Dependencies Replacement for Runlevels

[Patrick Donnelly-3]

Right now scripts are required to assign a runlevel for their scripts
to enforce an ordered execution of a group of scripts during an NSE
scan. As an example, smb-brute.nse uses a runlevel of 0.5 so it runs
before other smb-* scripts. This allows the other smb-* scripts to
utilize the results from the smb-brute.nse script. Unfortunately, it
can be difficult to identify the dependencies between these scripts
(specified loosely via runlevels) and impossible to enforce a
dependency (since scripts are not aware of what other scripts are
running). Enforcing a dependency means that we do not run our script
if we are missing a dependency (or, we abort scanning altogether
because a dependency is missing).

I have created a patch to NSE that replaces runlevels with a table of
dependencies that clearly outlines what other scripts the script
depends on. The table is of the form:

dependences = {"script1", script2", ...}

Runlevels become an internal representation of the order of scripts
that are generated by the dependencies. If a dependency is not present
in the current group of scripts then an error will be raised noting
the missing dependency. Alternatively, you can use the new command
line option --script-autoadd to automatically add dependencies to the
current group of scripts (this can potentially add dangerous scripts
and therefore is not the default).

We also have weak dependencies that specify scripts that the script
should run after but are not required for its execution. Its form is
the same as the dependencies table above.

The user will still see what the current runlevel is during the scan.
Additionally, they will now be aware of the number of runlevels:

NSE: Script scanning 127.0.0.1.
NSE: Starting runlevel 1 (of 3) scan.
Initiating NSE at 17:38
Completed NSE at 17:38, 0.00s elapsed
NSE: Starting runlevel 2 (of 3) scan.
Initiating NSE at 17:38
Completed NSE at 17:38, 0.00s elapsed
NSE: Starting runlevel 3 (of 3) scan.
Initiating NSE at 17:38
Completed NSE at 17:38, 0.00s elapsed
NSE: Script Scanning completed.

Another non-obvious benefit to explicit dependencies is we no longer
have scripts running in their own runlevel needlessly (reducing our
overall parallelism). Before, smb-brute would run by itself in
runlevel 0.5 when it could run alongside other unrelated scripts.

With respect to backwards compatibility, there is none. We ignore any
runlevel specification in the script. Explicit dependencies would be
required.

--
-Patrick Donnelly

"Let all men know thee, but no man know thee thoroughly: Men freely
ford that see the shallows."

- Benjamin Franklin

[dependency.5.patch]

---

Index: nmap.cc
===================================================================
--- nmap.cc (revision 16019)
+++ nmap.cc (working copy)
@@ -253,6 +253,7 @@
    "  --script-args=<n1=v1,[n2=v2,...]>: provide arguments to scripts\n"
        "  --script-trace: Show all data sent and received\n"
        "  --script-updatedb: Update the script database.\n"
+       "  --script-autoadd: Automatically add missing dependencies to run"
 #endif
        "OS DETECTION:\n"
        "  -O: Enable OS detection\n"
@@ -711,6 +712,8 @@
       {"script_updatedb", no_argument, 0, 0},
       {"script-args",required_argument,0,0},
       {"script_args",required_argument,0,0},
+      {"script-autoadd",no_argument,0,0},
+      {"script_autoadd",no_argument,0,0},
 #endif
       {"ip_options", required_argument, 0, 0},
       {"ip-options", required_argument, 0, 0},
@@ -756,6 +759,8 @@
               o.scripttrace = 1;
       } else if (optcmp(long_options[option_index].name, "script-updatedb") == 0){
               o.scriptupdatedb = 1;
+      } else if (optcmp(long_options[option_index].name, "script-autoadd") == 0){
+              o.scriptautoadd = 1;
       } else
 #endif
       if (optcmp(long_options[option_index].name, "max-os-tries") == 0) {
Index: NmapOps.cc
===================================================================
--- NmapOps.cc (revision 16019)
+++ NmapOps.cc (working copy)
@@ -305,6 +305,7 @@
   scriptversion = 0;
   scripttrace = 0;
   scriptupdatedb = 0;
+  scriptautoadd = 0;
 #endif
   memset(&sourcesock, 0, sizeof(sourcesock));
   sourcesocklen = 0;
Index: nse_main.cc
===================================================================
--- nse_main.cc (revision 16019)
+++ nse_main.cc (working copy)
@@ -236,6 +236,8 @@
   lua_setfield(L, -2, "default");
   lua_pushboolean(L, o.scriptversion == 1);
   lua_setfield(L, -2, "scriptversion");
+  lua_pushboolean(L, o.scriptautoadd == 1);
+  lua_setfield(L, -2, "scriptautoadd");
   lua_pushliteral(L, SCRIPT_ENGINE_LUA_DIR SCRIPT_ENGINE_DATABASE);
   lua_setfield(L, -2, "script_dbpath");
   lua_pushstring(L, o.scriptargs);
Index: NmapOps.h
===================================================================
--- NmapOps.h (revision 16019)
+++ NmapOps.h (working copy)
@@ -327,6 +327,7 @@
   int scriptversion;
   int scripttrace;
   int scriptupdatedb;
+  int scriptautoadd;
   void chooseScripts(char* argument);
   std::vector<std::string> chosenScripts;
 #endif
Index: nse_main.lua
===================================================================
--- nse_main.lua (revision 16019)
+++ nse_main.lua (working copy)
@@ -66,6 +66,8 @@
 
 local traceback = debug.traceback;
 
+local max = math.max;
+
 local byte = string.byte;
 local find = string.find;
 local format = string.format;
@@ -74,6 +76,7 @@
 local match = string.match;
 local sub = string.sub;
 
+local concat = table.concat;
 local insert = table.insert;
 local remove = table.remove;
 local sort = table.sort;
@@ -202,7 +205,6 @@
     if not self[rule] then return nil end -- No rule for this script?
     local file_closure = self.file_closure;
     local env = setmetatable({
-        runlevel = 1,
         filename = self.filename,
       }, {__index = _G});
     setfenv(file_closure, env);
@@ -224,7 +226,6 @@
       local thread = setmetatable({
         co = co,
         env = env,
-        runlevel = tonumber(rawget(env, "runlevel")) or 1,
         identifier = tostring(co),
         info = format("'%s' (%s)", self.short_basename, tostring(co));
         type = rule == "hostrule" and "host" or "port",
@@ -246,6 +247,8 @@
     description = "string",
     action = "function",
     categories = "table",
+    dependencies = "table",
+    weak_dependencies = "table",
   };
   -- script = Script.new(filename)
   -- Creates a new Script Class for the script.
@@ -264,7 +267,8 @@
     -- Give the closure its own environment, with global access
     local env = setmetatable({
       filename = filename,
-      runlevel = 1,
+      dependencies = {},
+      weak_dependencies = {},
     }, {__index = _G});
     setfenv(file_closure, env);
     local co = create(file_closure); -- Create a garbage thread
@@ -288,6 +292,16 @@
       assert(type(category) == "string",
         filename.." has non-string entries in the 'categories' array");
     end
+    -- Assert that dependencies is an array of strings
+    for i, dependency in ipairs(rawget(env, "dependencies")) do
+      assert(type(dependency) == "string",
+        filename.." has non-string entries in the 'dependencies' array");
+    end
+    -- Assert that weak_dependencies is an array of strings
+    for i, dependency in ipairs(rawget(env, "weak_dependencies")) do
+      assert(type(dependency) == "string",
+        filename.." has non-string entries in the 'weak_dependencies' array");
+    end
     -- Return the script
     return setmetatable({
       filename = filename,
@@ -303,7 +317,8 @@
       categories = rawget(env, "categories"),
       author = rawget(env, "author"),
       license = rawget(env, "license"),
-      runlevel = tonumber(rawget(env, "runlevel")) or 1,
+      dependencies = rawget(env, "dependencies"),
+      weak_dependencies = rawget(env, "weak_dependencies"),
       threads = {},
       selected_by_name = false,
     }, {__index = Script, __metatable = Script});
@@ -468,6 +483,69 @@
       end
     end
   end
+
+  -- calculate runlevels
+  local name_script = {};
+  for i, script in ipairs(chosen_scripts) do
+    assert(name_script[script.short_basename] == nil);
+    name_script[script.short_basename] = script;
+  end
+  local chain = {}; -- chain of script names
+  setmetatable(name_script, {
+    __index = function (t, k)
+      chain[#chain+1] = k;
+      if cnse.scriptautoadd then
+        local t, path = cnse.fetchfile_absolute(k);
+        if t == nil then -- omitted extension?
+          t, path = cnse.fetchfile_absolute(k..".nse");
+        end
+        if t == "file" then
+          local script = Script.new(path);
+          chosen_scripts[#chosen_scripts+1] = script;
+          name_script[script.short_basename] = script;
+          print_verbose(1,
+              "Script dependency '%s' in chain `%s` added automatically.",
+              script.filename, concat(chain, "->"));
+          chain[#chain] = nil;
+          return script;
+        else
+          error("could not locate script dependency in chain `"..
+              concat(chain, "->").."`");
+        end
+      else
+        error("missing dependency in chain `"..concat(chain, "->").."`\n"..
+            "(use --script-autoadd to automatically add missing dependencies)");
+      end
+    end
+  });
+  local function calculate_runlevel (script)
+    chain[#chain+1] = script.short_basename;
+    if script.runlevel == false then -- circular dependency
+      error("circular dependency in chain `"..concat(chain, "->").."`");
+    else
+      script.runlevel = false; -- placeholder
+    end
+    local runlevel = 1;
+    for i, dependency in ipairs(script.dependencies) do
+      local s = name_script[dependency];
+      local r = tonumber(s.runlevel) or calculate_runlevel(s);
+      runlevel = max(runlevel, r+1);
+    end
+    for i, weak_dependency in ipairs(script.weak_dependencies) do
+      local s = rawget(name_script, weak_dependency);
+      if s then
+        local r = tonumber(s.runlevel) or calculate_runlevel(s);
+        runlevel = max(runlevel, r+1);
+      end
+    end
+    chain[#chain] = nil;
+    script.runlevel = runlevel;
+    return runlevel;
+  end
+  for i, script in ipairs(chosen_scripts) do
+    local _ = script.runlevel or calculate_runlevel(script);
+  end
+
   return chosen_scripts;
 end
 
@@ -743,7 +821,8 @@
 
   sort(runlevels);
   for i, runlevel in ipairs(runlevels) do
-    print_verbose(1, "Starting runlevel %g scan", runlevel);
+    print_verbose(1, "Starting runlevel %u (of %u) scan.", runlevel,
+        #runlevels);
     run(threads[runlevel]);
   end
 

---

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Ron (list)]

Patrick Donnelly wrote:

> Right now scripts are required to assign a runlevel for their scripts
> to enforce an ordered execution of a group of scripts during an NSE
> scan. As an example, smb-brute.nse uses a runlevel of 0.5 so it runs
> before other smb-* scripts. This allows the other smb-* scripts to
> utilize the results from the smb-brute.nse script. Unfortunately, it
> can be difficult to identify the dependencies between these scripts
> (specified loosely via runlevels) and impossible to enforce a
> dependency (since scripts are not aware of what other scripts are
> running). Enforcing a dependency means that we do not run our script
> if we are missing a dependency (or, we abort scanning altogether
> because a dependency is missing).
>
> I have created a patch to NSE that replaces runlevels with a table of
> dependencies that clearly outlines what other scripts the script
> depends on. The table is of the form:
>
> dependences = {"script1", script2", ...}
>
> Runlevels become an internal representation of the order of scripts
> that are generated by the dependencies. If a dependency is not present
> in the current group of scripts then an error will be raised noting
> the missing dependency. Alternatively, you can use the new command
> line option --script-autoadd to automatically add dependencies to the
> current group of scripts (this can potentially add dangerous scripts
> and therefore is not the default).
>
> We also have weak dependencies that specify scripts that the script
> should run after but are not required for its execution. Its form is
> the same as the dependencies table above.
>
> The user will still see what the current runlevel is during the scan.
> Additionally, they will now be aware of the number of runlevels:
>
> NSE: Script scanning 127.0.0.1.
> NSE: Starting runlevel 1 (of 3) scan.
> Initiating NSE at 17:38
> Completed NSE at 17:38, 0.00s elapsed
> NSE: Starting runlevel 2 (of 3) scan.
> Initiating NSE at 17:38
> Completed NSE at 17:38, 0.00s elapsed
> NSE: Starting runlevel 3 (of 3) scan.
> Initiating NSE at 17:38
> Completed NSE at 17:38, 0.00s elapsed
> NSE: Script Scanning completed.
>
> Another non-obvious benefit to explicit dependencies is we no longer
> have scripts running in their own runlevel needlessly (reducing our
> overall parallelism). Before, smb-brute would run by itself in
> runlevel 0.5 when it could run alongside other unrelated scripts.
>
> With respect to backwards compatibility, there is none. We ignore any
> runlevel specification in the script. Explicit dependencies would be
> required.

I tested out this patch and I think it's great. I'm in favour of it
being included (and will modify my scripts -- I suspect I'm the only one
using runlevels).

Ron

--
Ron Bowes
http://www.skullsecurity.org/
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Fyodor]

On Sun, Nov 08, 2009 at 05:45:19PM -0500, Patrick Donnelly wrote:
>
> I have created a patch to NSE that replaces runlevels with a table of
> dependencies that clearly outlines what other scripts the script
> depends on. The table is of the form:

Yes, that sounds like a much better approach!

-F
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[David Fifield]

On Sun, Nov 08, 2009 at 05:45:19PM -0500, Patrick Donnelly wrote:

> Right now scripts are required to assign a runlevel for their scripts
> to enforce an ordered execution of a group of scripts during an NSE
> scan. As an example, smb-brute.nse uses a runlevel of 0.5 so it runs
> before other smb-* scripts. This allows the other smb-* scripts to
> utilize the results from the smb-brute.nse script. Unfortunately, it
> can be difficult to identify the dependencies between these scripts
> (specified loosely via runlevels) and impossible to enforce a
> dependency (since scripts are not aware of what other scripts are
> running). Enforcing a dependency means that we do not run our script
> if we are missing a dependency (or, we abort scanning altogether
> because a dependency is missing).

This is a great idea. I think this patch should go in, but first, I want
to see what people think about some changes to the interface.

> I have created a patch to NSE that replaces runlevels with a table of
> dependencies that clearly outlines what other scripts the script
> depends on. The table is of the form:
>
> dependences = {"script1", script2", ...}

What do you think about using "depends" instead of "dependencies"?
"dependencies" is hard to type.

"dependencies" doesn't correspond to anything in the current system,
right? There's no way currently for one script to require the execution
of another? weak_dependencies is what we emulate using runlevels.

I think people will have trouble with the distinction between "strong"
and "weak" dependencies. How about using a name like "run_after" for
weak dependencies?

> Runlevels become an internal representation of the order of scripts
> that are generated by the dependencies. If a dependency is not present
> in the current group of scripts then an error will be raised noting
> the missing dependency. Alternatively, you can use the new command
> line option --script-autoadd to automatically add dependencies to the
> current group of scripts (this can potentially add dangerous scripts
> and therefore is not the default).

It took second to figure out what --script-autoadd is doing. At first I
thought it was going to modify the script files themselves and add a
line with dependencies inferred from their runlevel ("automatically add
dependencies to the current group of scripts"), but that's not it at
all. Really, it's this: If a strong dependency is not satisfied by the
scripts you list, you get an error. So, for example I made html-title
depend on http-auth:

./nmap --datadir . --script=html-title scanme.nmap.org -p 80
NSE: failed to initialize the script engine:
./nse_main.lua:516: missing dependency in chain `html-title->http-auth`
(use --script-autoadd to automatically add missing dependencies)

Using --script-autoadd will automatically fill in any dependencies of
scripts, so you won't get that error. It's like "apt-get install ..."
installs all the packages that are recursive dependencies of the one you
want.

./nmap --datadir . --script=html-title --script-autoadd scanme.nmap.org -p 80 -d
NSE: Script dependency './scripts/http-auth.nse' in chain `html-title->http-auth` added automatically.
NSE: Loaded 2 scripts for scanning.

My question is, do we need support for strong dependencies? I'm assuming
I'm correct in thinking that strong dependencies are a new future, and
that weak dependencies are equivalent to runlevels. If there is a use
case for strong dependencies I'm not against them, but I would like to
avoid having a --script-autoadd option.

My bias at the moment is to just have a "depends" or "run_after"
variable that works like the weak_dependencies you have proposed, and
not implement strong dependencies, but I'm willing to hear potential
uses for strong dependencies.

> We also have weak dependencies that specify scripts that the script
> should run after but are not required for its execution. Its form is
> the same as the dependencies table above.
>
> The user will still see what the current runlevel is during the scan.
> Additionally, they will now be aware of the number of runlevels:
>
> NSE: Script scanning 127.0.0.1.
> NSE: Starting runlevel 1 (of 3) scan.
> Initiating NSE at 17:38
> Completed NSE at 17:38, 0.00s elapsed
> NSE: Starting runlevel 2 (of 3) scan.
> Initiating NSE at 17:38
> Completed NSE at 17:38, 0.00s elapsed
> NSE: Starting runlevel 3 (of 3) scan.
> Initiating NSE at 17:38
> Completed NSE at 17:38, 0.00s elapsed
> NSE: Script Scanning completed.

This is good output.

> Another non-obvious benefit to explicit dependencies is we no longer
> have scripts running in their own runlevel needlessly (reducing our
> overall parallelism). Before, smb-brute would run by itself in
> runlevel 0.5 when it could run alongside other unrelated scripts.
>
> With respect to backwards compatibility, there is none. We ignore any
> runlevel specification in the script. Explicit dependencies would be
> required.

I can think of one thing that runlevels offer that explicit dependencies
don't. Say you have a brute force script, and you want it to run after
every script that can potentially find a login for you. For example,
telnet-brute might want to run after http-userdir-enum because the
latter can identify usernames. We could do this with runlevels by giving
a the login-finding scripts a low runlevel, but with explicit
dependencies every brute force script will need to know about every
login script. I suspect this is not easy to solve from a user interface
point of view. One idea I had was something like

weak_dependencies = {"*login"}

"*login" would stand for a class of scripts that would somehow signify
that they satisfy it. The runlevel assigned to the brute script would
have to be at least one more than the highest runlevel of any *login
script.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Ron (list)]

David Fifield wrote:
> My question is, do we need support for strong dependencies? I'm assuming
> I'm correct in thinking that strong dependencies are a new future, and
> that weak dependencies are equivalent to runlevels. If there is a use
> case for strong dependencies I'm not against them, but I would like to
> avoid having a --script-autoadd option.
Once http-spider.nse exists, I'd like to write some scripts that depend
on it. They wouldn't just be helped by http-spider.nse, but they'd
*require* its output to run.

> I can think of one thing that runlevels offer that explicit dependencies
> don't. Say you have a brute force script, and you want it to run after
> every script that can potentially find a login for you. For example,
> telnet-brute might want to run after http-userdir-enum because the
> latter can identify usernames. We could do this with runlevels by giving
> a the login-finding scripts a low runlevel, but with explicit
> dependencies every brute force script will need to know about every
> login script. I suspect this is not easy to solve from a user interface
> point of view. One idea I had was something like
>
> weak_dependencies = {"*login"}
>
> "*login" would stand for a class of scripts that would somehow signify
> that they satisfy it. The runlevel assigned to the brute script would
> have to be at least one more than the highest runlevel of any *login
> script.

Not sure if this really affect what you're saying, but you made me think
of it. There are sort of three types of auth* scripts:
1. Finding users
2. Finding passwords (based on users)
3. Using users/passwords to get more information

With the smb-* scripts, I sort of do 1 and 2 backwards -- you can't
necessarily get good user output in step (1), so I enumerate users in
step (3), but I abstracted the functions out and sort of combine it into
step (2). But that's besides the point -- for http-*, telnet-*, etc,
finding users before finding passwords which is before finding deeper
info is useful.

Also, usernames and such should carry across scripts (like snmp-*
scripts could use the telnet-* and ftp-* bruteforced accounts, perhaps?)

Authentication-based scripts are interesting, for sure.

Ron

--
Ron Bowes
http://www.skullsecurity.org/
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Ron (list)]

David Fifield wrote:
> My question is, do we need support for strong dependencies? I'm assuming
> I'm correct in thinking that strong dependencies are a new future, and
> that weak dependencies are equivalent to runlevels. If there is a use
> case for strong dependencies I'm not against them, but I would like to
> avoid having a --script-autoadd option.
Once http-spider.nse exists, I'd like to write some scripts that depend
on it. They wouldn't just be helped by http-spider.nse, but they'd
*require* its output to run.

> I can think of one thing that runlevels offer that explicit dependencies
> don't. Say you have a brute force script, and you want it to run after
> every script that can potentially find a login for you. For example,
> telnet-brute might want to run after http-userdir-enum because the
> latter can identify usernames. We could do this with runlevels by giving
> a the login-finding scripts a low runlevel, but with explicit
> dependencies every brute force script will need to know about every
> login script. I suspect this is not easy to solve from a user interface
> point of view. One idea I had was something like
>
> weak_dependencies = {"*login"}
>
> "*login" would stand for a class of scripts that would somehow signify
> that they satisfy it. The runlevel assigned to the brute script would
> have to be at least one more than the highest runlevel of any *login
> script.

Not sure if this really affect what you're saying, but you made me think
of it. There are sort of three types of auth* scripts:
1. Finding users
2. Finding passwords (based on users)
3. Using users/passwords to get more information

With the smb-* scripts, I sort of do 1 and 2 backwards -- you can't
necessarily get good user output in step (1), so I enumerate users in
step (3), but I abstracted the functions out and sort of combine it into
step (2). But that's besides the point -- for http-*, telnet-*, etc,
finding users before finding passwords which is before finding deeper
info is useful.

Also, usernames and such should carry across scripts (like snmp-*
scripts could use the telnet-* and ftp-* bruteforced accounts, perhaps?)

Authentication-based scripts are interesting, for sure.

Ron

--
Ron Bowes
http://www.skullsecurity.org/
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[David Fifield]

On Tue, Nov 10, 2009 at 09:25:16AM -0600, Ron wrote:

> David Fifield wrote:
> > My question is, do we need support for strong dependencies? I'm assuming
> > I'm correct in thinking that strong dependencies are a new future, and
> > that weak dependencies are equivalent to runlevels. If there is a use
> > case for strong dependencies I'm not against them, but I would like to
> > avoid having a --script-autoadd option.
>
> Once http-spider.nse exists, I'd like to write some scripts that depend
> on it. They wouldn't just be helped by http-spider.nse, but they'd
> *require* its output to run.

That's a pretty good example. How do you see your script being invoked.
I can think of a few options. Below when I say "list all scripts" I mean
you can name the scripts individually or by category with --script, or
have them selected automatically by being in the default category.

1. You have to list all scripts, including dependencies, or Nmap will
   stop with an error.
2. If a dependency for a script is not listed, that script just won't
   run even if you asked for it.
3. Nmap won't run if not all dependencies are listed, but there is an
   option to automatically include any required dependencies.
4. Same as #2, with an option to automatically add dependencies.
5. Required dependencies are always added automatically; you may have
   scripts run that you didn't ask for specifically.

> > I can think of one thing that runlevels offer that explicit dependencies
> > don't. Say you have a brute force script, and you want it to run after
> > every script that can potentially find a login for you. For example,
> > telnet-brute might want to run after http-userdir-enum because the
> > latter can identify usernames. We could do this with runlevels by giving
> > a the login-finding scripts a low runlevel, but with explicit
> > dependencies every brute force script will need to know about every
> > login script. I suspect this is not easy to solve from a user interface
> > point of view. One idea I had was something like
> >
> > weak_dependencies = {"*login"}
> >
> > "*login" would stand for a class of scripts that would somehow signify
> > that they satisfy it. The runlevel assigned to the brute script would
> > have to be at least one more than the highest runlevel of any *login
> > script.
>
> Not sure if this really affect what you're saying, but you made me think
> of it. There are sort of three types of auth* scripts:
> 1. Finding users
> 2. Finding passwords (based on users)
> 3. Using users/passwords to get more information
>
> With the smb-* scripts, I sort of do 1 and 2 backwards -- you can't
> necessarily get good user output in step (1), so I enumerate users in
> step (3), but I abstracted the functions out and sort of combine it into
> step (2). But that's besides the point -- for http-*, telnet-*, etc,
> finding users before finding passwords which is before finding deeper
> info is useful.
>
> Also, usernames and such should carry across scripts (like snmp-*
> scripts could use the telnet-* and ftp-* bruteforced accounts, perhaps?)

Yeah--we should have some standard place in the registry where we can
store such things.

I think solving the *login problem could use some more discussion, but I
don't want it to stand in the way of this patch being integrated.
Patrick's implementation of dependencies is better than runlevel, and
this limitation is academic because it wouldn't break anything we're
doing now. I think it's worth thinking about, though.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Ron (list)]

David Fifield wrote:

> On Tue, Nov 10, 2009 at 09:25:16AM -0600, Ron wrote:
> That's a pretty good example. How do you see your script being invoked.
> I can think of a few options. Below when I say "list all scripts" I mean
> you can name the scripts individually or by category with --script, or
> have them selected automatically by being in the default category.
>
> 1. You have to list all scripts, including dependencies, or Nmap will
>    stop with an error.
> 2. If a dependency for a script is not listed, that script just won't
>    run even if you asked for it.
> 3. Nmap won't run if not all dependencies are listed, but there is an
>    option to automatically include any required dependencies.
> 4. Same as #2, with an option to automatically add dependencies.
> 5. Required dependencies are always added automatically; you may have
>    scripts run that you didn't ask for specifically.

I'm not sure which is best, really.

Personally, I dislike the idea of running scripts the user didn't ask
for, unless the user specifically says "run extra scripts to meet
dependencies". Otherwise, they don't necessarily know what they're
getting themselves into. http-spider.nse will probably be a long and
slow script, so running it without notifying the user is a little sketchy.

I hope some day the scripting will be more tightly integrated into
Zenmap, and Zenmap can automate a lot of the dependency stuff. That's
probably a long way off, though.

> Yeah--we should have some standard place in the registry where we can
> store such things.
>
> I think solving the *login problem could use some more discussion, but I
> don't want it to stand in the way of this patch being integrated.
> Patrick's implementation of dependencies is better than runlevel, and
> this limitation is academic because it wouldn't break anything we're
> doing now. I think it's worth thinking about, though.
Agreed.

Ron

--
Ron Bowes
http://www.skullsecurity.org/
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Patrick Donnelly-3]

Hi David,

On Mon, Nov 9, 2009 at 6:06 PM, David Fifield <david@...> wrote:
>> I have created a patch to NSE that replaces runlevels with a table of
>> dependencies that clearly outlines what other scripts the script
>> depends on. The table is of the form:
>>
>> dependences = {"script1", script2", ...}
>
> What do you think about using "depends" instead of "dependencies"?
> "dependencies" is hard to type.

I'm not particularly attached to the word, but I was trying to follow
the "established" convention of nouns for field names in the script
(action, description, author, etc.). Doesn't really matter to me.

> "dependencies" doesn't correspond to anything in the current system,
> right? There's no way currently for one script to require the execution
> of another? weak_dependencies is what we emulate using runlevels.

Right.

> I think people will have trouble with the distinction between "strong"
> and "weak" dependencies. How about using a name like "run_after" for
> weak dependencies?

I personally would prefer keeping dependencies over "run_after". I
think the weak/strong distinction will be transparent to users and
will only surface when a dependency is missing, in which case I think
it is clear the user should add the offending dependency or remove the
dependent script from their script group.

>> Runlevels become an internal representation of the order of scripts
>> that are generated by the dependencies. If a dependency is not present
>> in the current group of scripts then an error will be raised noting
>> the missing dependency. Alternatively, you can use the new command
>> line option --script-autoadd to automatically add dependencies to the
>> current group of scripts (this can potentially add dangerous scripts
>> and therefore is not the default).
>
> It took second to figure out what --script-autoadd is doing. At first I
> thought it was going to modify the script files themselves and add a
> line with dependencies inferred from their runlevel ("automatically add
> dependencies to the current group of scripts"), but that's not it at
> all. Really, it's this: If a strong dependency is not satisfied by the
> scripts you list, you get an error. So, for example I made html-title
> depend on http-auth:
>
> ./nmap --datadir . --script=html-title scanme.nmap.org -p 80
> NSE: failed to initialize the script engine:
> ./nse_main.lua:516: missing dependency in chain `html-title->http-auth`
> (use --script-autoadd to automatically add missing dependencies)
>
> Using --script-autoadd will automatically fill in any dependencies of
> scripts, so you won't get that error. It's like "apt-get install ..."
> installs all the packages that are recursive dependencies of the one you
> want.

Right, sorry that wasn't clear enough.

> ./nmap --datadir . --script=html-title --script-autoadd scanme.nmap.org -p 80 -d
> NSE: Script dependency './scripts/http-auth.nse' in chain `html-title->http-auth` added automatically.
> NSE: Loaded 2 scripts for scanning.
>
> My question is, do we need support for strong dependencies? I'm assuming
> I'm correct in thinking that strong dependencies are a new future, and
> that weak dependencies are equivalent to runlevels. If there is a use
> case for strong dependencies I'm not against them, but I would like to
> avoid having a --script-autoadd option.

Strong dependencies would be the new future. I don't really know how
many scripts would use strong dependencies (right now) if they were
available. I think maybe one or two would. Right now we have scripts
using the information generated from previous scripts in a loose
fashion. It may be better if these scripts could rely on that
information being there through a dependency system.

> I can think of one thing that runlevels offer that explicit dependencies
> don't. Say you have a brute force script, and you want it to run after
> every script that can potentially find a login for you. For example,
> telnet-brute might want to run after http-userdir-enum because the
> latter can identify usernames. We could do this with runlevels by giving
> a the login-finding scripts a low runlevel, but with explicit
> dependencies every brute force script will need to know about every
> login script. I suspect this is not easy to solve from a user interface
> point of view. One idea I had was something like
>
> weak_dependencies = {"*login"}
>
> "*login" would stand for a class of scripts that would somehow signify
> that they satisfy it. The runlevel assigned to the brute script would
> have to be at least one more than the highest runlevel of any *login
> script.

Ron and I considered adding something like the above. I believe the
consensus was that we would wait for a script that clearly needs it
rather than add it right now. We don't actually have a unified
approach with our scripts with respect to host state/information
saving. As an example, we don't save the usernames we find in some
agreed upon "official" location. Until we do, scripts won't be relying
on unknown scripts that produce this information (a generic *-login
wildcard).

--
-Patrick Donnelly

"Let all men know thee, but no man know thee thoroughly: Men freely
ford that see the shallows."

- Benjamin Franklin
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Fyodor]

On Tue, Nov 10, 2009 at 09:25:16AM -0600, Ron wrote:
> David Fifield wrote:
>
> Once http-spider.nse exists, I'd like to write some scripts that depend
> on it. They wouldn't just be helped by http-spider.nse, but they'd
> *require* its output to run.

I'm wondering if a library would be better for this sort of need than
"strong dependencies"?  Let's look at these two options in the case of
http-spider:

==As a spidering library:

 o Your script which depends on spidering can pass an argument saying
   what type of content you're interested in (e.g. do you care about
   images at all?  If not, no point in a script like sql-injection
   downloading them)

 o You can pass arguments specifying how deeply you want to spider
   (e.g. stop after 1,000 requests or 5 minutes, whichever comes first)

 o Depending on how the spidering library is architected, you may be
   able to control it interactively (e.g. have it call you back with each
   resource so you can deal with it one page at a time and perhaps
   specify which embedded links you want to (or do not want to) spider
   recursively.

 o The library doesn't request anything if none of the scripts end up
   needing it.  And even if it is needed, it doesn't end up running
   until that point where it will need to be used (because it gets
   called then).

 o The same script or other scripts can call it later if more
   information is needed.  For example, sql-injection may not care
   about images, but http-mirror might.  So http-mirror will call it
   with different arguments and the pages will come from our cache
   while the images will be fetched for the first time.

 o I'm assuming that the worker processes (Patrick's other neat
   pending patch) could be used by a library just as they can by a
   script.

==As a script

 o I guess the script would just have to do something generic like
   fetch a bunch of pages (including images, I suppose), and then stop
   at some point and hope it has gathered whatever the scripts which
   depend on it need.  It can see user-specified NSE arguments, just
   as a library could, but doesn't know anything about the
   requirements of the scripts which depend on it.

For these reasons, it seems to me that a spidering NSE library would
be better than a script, even if we had strong dependencies.

Cheers,
Fyodor
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Fyodor]

On Mon, Nov 09, 2009 at 04:06:18PM -0700, David Fifield wrote:
> On Sun, Nov 08, 2009 at 05:45:19PM -0500, Patrick Donnelly wrote:
>
> This is a great idea. I think this patch should go in, but first, I want
> to see what people think about some changes to the interface.

I agree--this is a nice improvement!

> My question is, do we need support for strong dependencies? I'm assuming
> I'm correct in thinking that strong dependencies are a new future, and
> that weak dependencies are equivalent to runlevels. If there is a use
> case for strong dependencies I'm not against them, but I would like to
> avoid having a --script-autoadd option.

I could be missing something, but I think libraries can easily fill
the need of a "strong dependency" script with less complication.  The
first time a library function is called it can (take a mutex to insure
this isn't happening in parallel and) do the requested work for the
caller and also save any results for future callers if desired.  This
allows the script to better communicate what it wants (e.g. it can
specify parameters) and it also avoids users having to worry about
specifying scripts which are only needed because some other script
depends on them.  Users should only have to specify what they want
done (scripts) and shouldn't need to worry about their internal
implementation.

> I think people will have trouble with the distinction between "strong"
> and "weak" dependencies. How about using a name like "run_after" for
> weak dependencies?

weak_dependencies is a bit of a mouthful.  run_after sounds a bit
better.  If we only support one kind of dependencies, we could use the
"deps" keyword for them.  It is short and already has an established
meaning of "dependencies" in some circles.  It might be a bit
confusing if we're referring to the optional (weak) dependencies.  So
I'm fully open to other names.

> My bias at the moment is to just have a "depends" or "run_after"
> variable that works like the weak_dependencies you have proposed, and
> not implement strong dependencies, but I'm willing to hear potential
> uses for strong dependencies.

Yeah, I think we should be sure that strong dependencies are desirable
before adding them.

> For example,
> telnet-brute might want to run after http-userdir-enum because the
> latter can identify usernames. We could do this with runlevels by giving
> a the login-finding scripts a low runlevel, but with explicit
> dependencies every brute force script will need to know about every
> login script. I suspect this is not easy to solve from a user interface
> point of view. One idea I had was something like
>
> weak_dependencies = {"*login"}
>
> "*login" would stand for a class of scripts that would somehow signify
> that they satisfy it. The runlevel assigned to the brute script would
> have to be at least one more than the highest runlevel of any *login
> script.

That makes sense.  We might not need that just yet, but I think we
should not settle on a solution which doesn't allow us the capability
to add such a feature in the future.  I think the RPM package manager
handles this in a decent way.  You can "require" a specific package
name, or a more general functionality which other packages can provide
using the "provides" directive.

Cheers,
-F
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Fyodor]

On Tue, Nov 10, 2009 at 09:51:32AM -0700, David Fifield wrote:

> On Tue, Nov 10, 2009 at 09:25:16AM -0600, Ron wrote:
> > David Fifield wrote:
>
> That's a pretty good example. How do you see your script being invoked.
> I can think of a few options. Below when I say "list all scripts" I mean
> you can name the scripts individually or by category with --script, or
> have them selected automatically by being in the default category.
>
> 1. You have to list all scripts, including dependencies, or Nmap will
>    stop with an error.
> 2. If a dependency for a script is not listed, that script just won't
>    run even if you asked for it.
> 3. Nmap won't run if not all dependencies are listed, but there is an
>    option to automatically include any required dependencies.
> 4. Same as #2, with an option to automatically add dependencies.
> 5. Required dependencies are always added automatically; you may have
>    scripts run that you didn't ask for specifically.

I'm not sure that we need "strong dependencies", but if we do then I
prefer #5 so that Nmap adds them automatically.  The reasons for this
are:

1) Users should not have to know the implementation details of the
   scripts they want to run.  If the sql-injection script uses an
   http-spider script rather than a library or handling it internally,
   users should not have to know that and recite it in their command
   line.  But Nmap can always note in verbose mode if it is pulling in
   a dependency.

2) If the dependencies are added implicitly, Nmap will only include
   the ones actually needed.  If users are responsible, they may load
   up the command line with dependency scripts, some of which might
   not even be needed for a run.  It would be a waste to have the
   likes of http-spider run and use a lot of bandwidth if it turns out
   that no script even used the results.

Of course we also wouldn't want someone to specify one safe script and
have it specify a dependency which causes a dangerous script to run
and do something completely different.  But that should never happen
in the first place.  Scripts should not depend on another script which
does something likely to be unexpected to the user.

Of course we can avoid the whole issue if we don't allow required
dependencies in the first place.

Cheers,
Fyodor
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Ron (list)]

Fyodor wrote:

> On Tue, Nov 10, 2009 at 09:25:16AM -0600, Ron wrote:
> I'm wondering if a library would be better for this sort of need than
> "strong dependencies"?  Let's look at these two options in the case of
> http-spider:
>
> ==As a spidering library:
>
>  o Your script which depends on spidering can pass an argument saying
>    what type of content you're interested in (e.g. do you care about
>    images at all?  If not, no point in a script like sql-injection
>    downloading them)
>
>  o You can pass arguments specifying how deeply you want to spider
>    (e.g. stop after 1,000 requests or 5 minutes, whichever comes first)
>
>  o Depending on how the spidering library is architected, you may be
>    able to control it interactively (e.g. have it call you back with each
>    resource so you can deal with it one page at a time and perhaps
>    specify which embedded links you want to (or do not want to) spider
>    recursively.
>
>  o The library doesn't request anything if none of the scripts end up
>    needing it.  And even if it is needed, it doesn't end up running
>    until that point where it will need to be used (because it gets
>    called then).
>
>  o The same script or other scripts can call it later if more
>    information is needed.  For example, sql-injection may not care
>    about images, but http-mirror might.  So http-mirror will call it
>    with different arguments and the pages will come from our cache
>    while the images will be fetched for the first time.
>
>  o I'm assuming that the worker processes (Patrick's other neat
>    pending patch) could be used by a library just as they can by a
>    script.
>
> ==As a script
>
>  o I guess the script would just have to do something generic like
>    fetch a bunch of pages (including images, I suppose), and then stop
>    at some point and hope it has gathered whatever the scripts which
>    depend on it need.  It can see user-specified NSE arguments, just
>    as a library could, but doesn't know anything about the
>    requirements of the scripts which depend on it.
>
> For these reasons, it seems to me that a spidering NSE library would
> be better than a script, even if we had strong dependencies.
>
> Cheers,
> Fyodor

Although I agree that implementing strong dependencies as libraries is
possible, I think that 'strong dependencies' (or whatever you want to
call them) is a better solution in terms of code complexity. I can see
having a function that downloads certain pages depending on the
arguments, and blocking if other scripts are using it, getting
complicated fast. Having 'worker' scripts that get the information
beforehand seems conceptually nicer to me.

By your logic, it would make sense to eliminate all dependencies (weak
and strong) in favour of libraries. Is there a difference between a weak
and strong dependency that makes strong dependencies a better choices
for libraries and weak dependencies better for other scripts?

Also, having both weak and strong dependencies also gives a nice sense
of symmetry to me. I like symmetry. :)

Ron

--
Ron Bowes
http://www.skullsecurity.org/
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Patrick Donnelly-3]

Hi Fyodor,

On Tue, Nov 10, 2009 at 10:20 PM, Fyodor <fyodor@...> wrote:

>> My question is, do we need support for strong dependencies? I'm assuming
>> I'm correct in thinking that strong dependencies are a new future, and
>> that weak dependencies are equivalent to runlevels. If there is a use
>> case for strong dependencies I'm not against them, but I would like to
>> avoid having a --script-autoadd option.
>
> I could be missing something, but I think libraries can easily fill
> the need of a "strong dependency" script with less complication.  The
> first time a library function is called it can (take a mutex to insure
> this isn't happening in parallel and) do the requested work for the
> caller and also save any results for future callers if desired.  This
> allows the script to better communicate what it wants (e.g. it can
> specify parameters) and it also avoids users having to worry about
> specifying scripts which are only needed because some other script
> depends on them.  Users should only have to specify what they want
> done (scripts) and shouldn't need to worry about their internal
> implementation.

It's admittedly difficult (for me) to identify a scenario where a
script (strong) dependency makes more sense than the requiring of a
library. Despite thinking about this a lot, I haven't decided which is
better; however, my instinct is that libraries are inappropriate for
doing the tasks that are better encapsulated in a script (fetching
possible user names for later brute forcing) -- while I am making the
http-spider functionality in a library, there will still be a script
that actually runs the spider. Anyway, whatever you guys decide I'm ok
with.

>> I think people will have trouble with the distinction between "strong"
>> and "weak" dependencies. How about using a name like "run_after" for
>> weak dependencies?
> weak_dependencies is a bit of a mouthful. run_after sounds a bit better.
> If we only support one kind of dependencies, we could use the "deps"
> keyword for them.  It is short and already has an established meaning
> of "dependencies" in some circles.  It might be a bit confusing if
> we're referring to the optional (weak) dependencies.  So I'm fully
> open to other names.

The names (dependencies & weak_dependencies) are long but so are
others (description). These names are written only once and need not
be specified in every script (the default is an empty dependencies
table). Here I think full names are fine. Also, there is no reason for
a script to reference the dependencies table in their code so brevity
doesn't gain us anything.

--
-Patrick Donnelly

"Let all men know thee, but no man know thee thoroughly: Men freely
ford that see the shallows."

- Benjamin Franklin
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Ron (list)]

Patrick Donnelly wrote:

> Right now scripts are required to assign a runlevel for their scripts
> to enforce an ordered execution of a group of scripts during an NSE
> scan. As an example, smb-brute.nse uses a runlevel of 0.5 so it runs
> before other smb-* scripts. This allows the other smb-* scripts to
> utilize the results from the smb-brute.nse script. Unfortunately, it
> can be difficult to identify the dependencies between these scripts
> (specified loosely via runlevels) and impossible to enforce a
> dependency (since scripts are not aware of what other scripts are
> running). Enforcing a dependency means that we do not run our script
> if we are missing a dependency (or, we abort scanning altogether
> because a dependency is missing).
>
> I have created a patch to NSE that replaces runlevels with a table of
> dependencies that clearly outlines what other scripts the script
> depends on. The table is of the form:
>
> dependences = {"script1", script2", ...}
>
> Runlevels become an internal representation of the order of scripts
> that are generated by the dependencies. If a dependency is not present
> in the current group of scripts then an error will be raised noting
> the missing dependency. Alternatively, you can use the new command
> line option --script-autoadd to automatically add dependencies to the
> current group of scripts (this can potentially add dangerous scripts
> and therefore is not the default).
>
> We also have weak dependencies that specify scripts that the script
> should run after but are not required for its execution. Its form is
> the same as the dependencies table above.
>
> The user will still see what the current runlevel is during the scan.
> Additionally, they will now be aware of the number of runlevels:
>
> NSE: Script scanning 127.0.0.1.
> NSE: Starting runlevel 1 (of 3) scan.
> Initiating NSE at 17:38
> Completed NSE at 17:38, 0.00s elapsed
> NSE: Starting runlevel 2 (of 3) scan.
> Initiating NSE at 17:38
> Completed NSE at 17:38, 0.00s elapsed
> NSE: Starting runlevel 3 (of 3) scan.
> Initiating NSE at 17:38
> Completed NSE at 17:38, 0.00s elapsed
> NSE: Script Scanning completed.
>
> Another non-obvious benefit to explicit dependencies is we no longer
> have scripts running in their own runlevel needlessly (reducing our
> overall parallelism). Before, smb-brute would run by itself in
> runlevel 0.5 when it could run alongside other unrelated scripts.
>
> With respect to backwards compatibility, there is none. We ignore any
> runlevel specification in the script. Explicit dependencies would be
> required.

We talked about wildcard dependencies somewhere in this thread, and how
there wasn't much of a case for using them, but I thought of something.

smb-security-mode.nse prints out the user that was performing all the
tests. To get a good reading, it should run after at least one script.
Therefore, smb-security-mode.nse sort of has a weak dependency on smb-*.
So there's a use case for it.

Right now, I'm solving it with "runlevel = 1.01", but that's hacky.


--
Ron Bowes
http://www.skullsecurity.org/
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[David Fifield]

On Fri, Nov 13, 2009 at 05:58:58PM -0600, Ron wrote:

> Patrick Donnelly wrote:
> > Right now scripts are required to assign a runlevel for their scripts
> > to enforce an ordered execution of a group of scripts during an NSE
> > scan. As an example, smb-brute.nse uses a runlevel of 0.5 so it runs
> > before other smb-* scripts. This allows the other smb-* scripts to
> > utilize the results from the smb-brute.nse script. Unfortunately, it
> > can be difficult to identify the dependencies between these scripts
> > (specified loosely via runlevels) and impossible to enforce a
> > dependency (since scripts are not aware of what other scripts are
> > running). Enforcing a dependency means that we do not run our script
> > if we are missing a dependency (or, we abort scanning altogether
> > because a dependency is missing).
> >
> > I have created a patch to NSE that replaces runlevels with a table of
> > dependencies that clearly outlines what other scripts the script
> > depends on. The table is of the form:
> >
> > dependences = {"script1", script2", ...}
> >
> > Runlevels become an internal representation of the order of scripts
> > that are generated by the dependencies. If a dependency is not present
> > in the current group of scripts then an error will be raised noting
> > the missing dependency. Alternatively, you can use the new command
> > line option --script-autoadd to automatically add dependencies to the
> > current group of scripts (this can potentially add dangerous scripts
> > and therefore is not the default).
> >
> > We also have weak dependencies that specify scripts that the script
> > should run after but are not required for its execution. Its form is
> > the same as the dependencies table above.
> >
> > The user will still see what the current runlevel is during the scan.
> > Additionally, they will now be aware of the number of runlevels:
> >
> > NSE: Script scanning 127.0.0.1.
> > NSE: Starting runlevel 1 (of 3) scan.
> > Initiating NSE at 17:38
> > Completed NSE at 17:38, 0.00s elapsed
> > NSE: Starting runlevel 2 (of 3) scan.
> > Initiating NSE at 17:38
> > Completed NSE at 17:38, 0.00s elapsed
> > NSE: Starting runlevel 3 (of 3) scan.
> > Initiating NSE at 17:38
> > Completed NSE at 17:38, 0.00s elapsed
> > NSE: Script Scanning completed.
> >
> > Another non-obvious benefit to explicit dependencies is we no longer
> > have scripts running in their own runlevel needlessly (reducing our
> > overall parallelism). Before, smb-brute would run by itself in
> > runlevel 0.5 when it could run alongside other unrelated scripts.
> >
> > With respect to backwards compatibility, there is none. We ignore any
> > runlevel specification in the script. Explicit dependencies would be
> > required.
>
> We talked about wildcard dependencies somewhere in this thread, and how
> there wasn't much of a case for using them, but I thought of something.
>
> smb-security-mode.nse prints out the user that was performing all the
> tests. To get a good reading, it should run after at least one script.
> Therefore, smb-security-mode.nse sort of has a weak dependency on smb-*.
> So there's a use case for it.
>
> Right now, I'm solving it with "runlevel = 1.01", but that's hacky.

I didn't mean "*login" is a wildcard matching script names. It just
stands for any syntax that could be used to stand for a class of
scripts. It could be [login] or $auth$ or whatever, just to indicate
that a script is capable of finding accounts that other scripts might
want to use. Your use of runlevel = 1.01 is exactly what would need a
replacement when we change to a dependency system.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[Fyodor]

On Fri, Nov 13, 2009 at 06:29:06PM -0700, David Fifield wrote:

> On Fri, Nov 13, 2009 at 05:58:58PM -0600, Ron wrote:
> > Patrick Donnelly wrote:
> >
> > smb-security-mode.nse prints out the user that was performing all the
> > tests. To get a good reading, it should run after at least one script.
> > Therefore, smb-security-mode.nse sort of has a weak dependency on smb-*.
> > So there's a use case for it.
> >
> > Right now, I'm solving it with "runlevel = 1.01", but that's hacky.
>
> I didn't mean "*login" is a wildcard matching script names. It just
> stands for any syntax that could be used to stand for a class of
> scripts. It could be [login] or $auth$ or whatever, just to indicate
> that a script is capable of finding accounts that other scripts might
> want to use. Your use of runlevel = 1.01 is exactly what would need a
> replacement when we change to a dependency system.

I realize that matching scripts with wildcards in the dependency
specification is not exactly what you had in mind, but it doesn't
sound like a bad idea to me.  At first it seemed hackish, but it does
simplify things as you don't need to add the likes of
"provides=brute-force" lines in *-brute.  If some scripts have names
which aren't conducive to this sort of globbing, they can always be
listed explicitly.  So my first reaction is actually to prefer this
sort of script ID (same as the filename) globbing to a more formalized
system of provides/requires.

Cheers,
Fyodor
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/

[David Fifield]

On Sun, Nov 08, 2009 at 05:45:19PM -0500, Patrick Donnelly wrote:

> Right now scripts are required to assign a runlevel for their scripts
> to enforce an ordered execution of a group of scripts during an NSE
> scan. As an example, smb-brute.nse uses a runlevel of 0.5 so it runs
> before other smb-* scripts. This allows the other smb-* scripts to
> utilize the results from the smb-brute.nse script. Unfortunately, it
> can be difficult to identify the dependencies between these scripts
> (specified loosely via runlevels) and impossible to enforce a
> dependency (since scripts are not aware of what other scripts are
> running). Enforcing a dependency means that we do not run our script
> if we are missing a dependency (or, we abort scanning altogether
> because a dependency is missing).
>
> I have created a patch to NSE that replaces runlevels with a table of
> dependencies that clearly outlines what other scripts the script
> depends on.

Patrick, please make a branch for this patch. I want it to be merged,
but it should be done all at once with the addition of dependency tables
to the scripts themselves.

Here is what Fyodor and I would like you to do in the branch. First,
remove strong dependencies and make "dependencies" the keyword for weak
dependencies. The idea behind that is that weak dependencies are going
to be the common case, and so should have a shorter name. If a need is
demonstrated for strong dependencies, then they could be re-added with
the keyword strong_dependencies or strict_dependencies or something like
that.

Next, add dependency tables to scripts that need them. Ron, since your
scripts are the biggest users of runlevels, would you figure out their
dependency tables? There are four different runlevels in use now: 0.5,
1.0, 1.01, and 2.0. It will be a matter of, for example, looking at all
the runlevel 1.0 scripts, seeing which ones really depend on the
runlevel 0.5 scripts, and so on.

Finally, replace the description of runlevels in scripting.xml with a
description of how dependencies work. You did such a good job
introducing this patch to the mailing list that you can use your post
and its examples as a model.

What do you think of this plan? Is moving from both strong and weak
dependencies to only weak dependencies a reasonable idea? We just want
to see a script that needs strong dependencies before adding the
feature.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/