batfish / batfish Goto Github PK

Provide a way to supply dynamic bgp information attached to a testrig/environment.

• In first pass, support output of show ip bgp on Arista EOS

Status: FAILURE
Question: null
{
"class" : "org.batfish.common.BatfishException$BatfishStackTrace",
"contents" : [
"org.batfish.common.BatfishException: Batfish job failed",
" at org.batfish.main.Driver$1.run(Driver.java:302)",
"Caused by: org.batfish.common.BatfishException: unimplemented",
" at org.batfish.datamodel.routing_policy.expr.NamedAsPathSet.matches(NamedAsPathSet.java:31)",
" at org.batfish.datamodel.routing_policy.expr.MatchAsPath.evaluate(MatchAsPath.java:27)",
" at org.batfish.datamodel.routing_policy.statement.If.execute(If.java:35)",
" at org.batfish.datamodel.routing_policy.RoutingPolicy.call(RoutingPolicy.java:38)",
" at org.batfish.datamodel.routing_policy.RoutingPolicy.process(RoutingPolicy.java:78)",
" at org.batfish.bdp.Node.propagateBgpRoutes(Node.java:801)",
" at org.batfish.bdp.BdpDataPlanePlugin.lambda$computeFixedPoint$15(BdpDataPlanePlugin.java:369)",
" at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)",
" at java.util.TreeMap$ValueSpliterator.forEachRemaining(TreeMap.java:2893)",
" at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)",
" at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)",
" at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)",
" at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)",
" at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)",
" at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)",
" at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)"
]
}

Need to add SNAT configuration to data model, includes addressing (e.g. ip nat pool), and NAT ACLs

bgp additional-paths select all
bgp additional-paths send receive
neighbor xxx advertise additional-paths all

And then it crashes when running a query. Ideally, checks should be done during both environment creation and when a non-existing interface in encountered during analysis.

The following exception occurs when running reduced reachability query after creating an environment with non-existing interface

org.batfish.common.BatfishException: Batfish job failed
at org.batfish.main.Driver$1.run(Driver.java:388)
Caused by: java.lang.NullPointerException
at org.batfish.main.Batfish.processInterfaceBlacklist(Batfish.java:3456)
at org.batfish.main.Batfish.loadConfigurations(Batfish.java:2452)
at org.batfish.bdp.BdpDataPlanePlugin.computeDataPlane(BdpDataPlanePlugin.java:259)
at org.batfish.main.Batfish.computeDataPlane(Batfish.java:853)
at org.batfish.main.Batfish.initQuestionEnvironment(Batfish.java:2170)
at org.batfish.main.Batfish.initQuestionEnvironments(Batfish.java:2183)
at org.batfish.main.Batfish.answer(Batfish.java:493)
at org.batfish.main.Batfish.run(Batfish.java:3858)
at org.batfish.main.Driver$1.run(Driver.java:356)

Provide standard way to include per-node routing tables with testrig. These should be usable for route validation as well as external bgp announcement inference.

While implemented in backend of z3 pipeline and in HeaderSpace, ecn, dscp, and tcpflags are not specifiable in either Reachability base or template questions.

The grammar files are helpful, but i don't know how to compile them correctly with antlr4.
projects/batfish/src/org/batfish/grammar/

Using the question type.

Similar to old flow sink system, user should be able to supply metadata in testrig specifying which interfaces are to be considered external vs internal for analysis purposes.

org.batfish.common.BatfishException: Unsupported vendor for OSPF inter-area summary administrative cost: FLAT_JUNIPER

if we add test to the arguments to batfish_build_all.sh it will run the test target in every ant build.xml. This will make sure the unit tests that exist (if any) will run.

We'll need to add the test ant target everywhere.

On Cisco, a peer-group must be defined before it can be activated. Batfish currently enforces this.
On Arista, activating a non-existent peer-group creates that peer-group. Batfish should detect configuration format and behave accordingly.
Furthermore, activating a non-existent peer-group on Cisco should generate a warning rather than failure.

Legal on Arista / illegal on Cisco:

router bgp 1
 address-family ipv4
  neighbor UNDEFINED_PEER_GROUP activate

ntp server vrf some-vrf 1.2.3.4 source loopback 0

When executing a query, batfish.standard() creates many NOD jobs (one per ingress), each of which ends up completely regenerating the NOD program even though the program is identical. So in the example rig, we are generating the program 15 times (instead of one)

One way to mitigate this is to add caching to the z3.Synthesizer to reuse the same statements and program -- since all the jobs are using the same synthesizer instance

Add support for SNAT packet transformation in reachability analysis as well as virtual traceroute. Depends on #106.

Currently only configuration files in top level of testrig configs folder are read. Instead, the whole tree under configs should be read.

Current js client relies on the underlying structure of the batfish directory to get the text of each config file. An API call to coordinator currently exists to get a list of all config files for a given testrig, but not the file text. It would be beneficial to have a filetext element in the answer of that API call.

We've got some fairly big file descriptor leaks due to use of various Stream<Path> APIs rooted in Files. We need to audit the codebase for cleaning this up.

A sample fix:

@@ -780,9 +783,13 @@ public class Batfish extends PluginConsumer implements AutoCloseable, IBatfish {
          throw new CleanBatfishException(
                "Missing compiled vendor-independent configurations for this test-rig\n");
       }
-      else if (CommonUtil.list(path).count() == 0) {
-         throw new CleanBatfishException(
-               "Nothing to do: Set of vendor-independent configurations for this test-rig is empty\n");
+      else {
+         try (Stream<Path> paths = CommonUtil.list(path)) {
+            if (paths.count() == 0) {
+               throw new CleanBatfishException(
+                     "Nothing to do: Set of vendor-independent configurations for this test-rig is empty\n");
+            }
+         }
       }
    }
 

[Geeze I hope there's a more concise way to do this.]

Run tests from the root of the batfish repository
allinone -cmdfile tests/java/commands

There are different command files
./demos/java/commands
./test_rigs/parsing-tests/commands
./tests/basic/commands
./tests/ui-focused/commands

Record instances of spanning-tree portfast at interface-level and top-level.
On Arista, this must be configured per-interface.
On Cisco, the default can be configured at top-level.

• Change to an imported grammar file should trigger regeneration of grammar importing it
  Currently, only master parser/lexer grammar changes trigger a rebuild. Fixing this is HIGHEST priority

• .tokens files should only be generated in target
  The .tokens files are now being generated in the src/main/antlr4/... directories. These should ideally not be produced in a src folder.

• Only recompile relevant java files when grammars are regenerated
  After any MASTER grammar file is changed, it appears that ALL java files are recompiled - rather than just the ones that have changed (or rely on ones that have changed). While this will certainly work, it seems heavy-handed. Please see behavior of old ant build.xml for a lighter-weight method.

• Parallel processing of grammars
  Grammars are currently processed one at a time. When we used ant, all out-of-date lexers were generated in parallel, and then all out-of-date parsers were generated in parallel. While we could replicate this behavior, I'd like to do better:
  Each grammar should be its own task. Each parser task should depend on its corresponding lexer task. All such tasks should be executed in parallel to the extent feasible.

We'd like to parse and retain the helper address for DHCP servers.

In the RPM/DEB building scripts, we install batfish and the batfish coordinator as system services. Right now they are not linked, so we have to restart each service independently.

It would be good to link them, so that we can restart a single service and have them all restart.

Right now the logic in Batfish#preprocessQuestion is complicated/brittle and also incomplete (specifically for the case of CompositeQuestion, which might have question-valued variables). We'd like to rewrite this code to be simpler, better tested, and potentially more powerful.

Specifically,

• add comprehensive tests.
• support composite question with question-valued variables.
• consider adding much more functionality than simple variable replacement like we have now.

The user should be warned (log level WARN instead of INFO) when load-questions fails to find any JSON template questions.

Assert question on jsonpaths

Arguments supplied to template variables should be validated by batfish-client before being sent off to coordinator.

Currently ip address and network specified in e.g.:

interface Vlan1
 ip address virtual 192.168.0.1/24

is completely ignored.
Support should be added throughout pipeline.

Testrigs should have a creation timestamp somewhere.

Every question answer should have a one-line summary field.

Currently, there is only one option for coordinator that sets the bind address for both pool manager and work manager: -servicehost
These should be split into two options: -poolbindhost and workbindhost

After fixing BGP in BDP, the example testrig does not actually perform as indicated. A new example testrig called example2 has been added. It uses bgp additional-path to recover the expected BGP multipath behavior in presence of route-reflectors. This should supplant the original example testrig. Depends on #133

automatically trigger parsing a testrig when a question is asked, if the testrig hasn't already been parsed? (like we do for data plane generation.)

with asynch clients, there may be testrigs that have not been parsed (e.g., because the client was killed right after the upload happened). then, we ask question of this testrig, we currently get:
FATAL ERROR: Missing compiled vendor-independent configurations for this test-rig)

we can do more complicated logic, e.g., the UI checks readiness before issuing the command to answer, but that seems unnecessary front end complexity if the backend can handle things.

When the travis build script .travis/build.sh crashes during the demo portion on a machine without z3 installed, the diffs of other refs are not shown.

It looks like BGP receivers in Batfish detect AS-path loops only by looking for their AS number in the first position of the AS-path. That would allow for loopy AS-paths such as A-X-Y-A-Z. In reality, such paths will not be allowed.

Pretty-print output referencing configuration structures should also contain marked up file and line information to locate those structures.

Currently the only way to see parsing ERRORs (not warnings) is to use options such that the parsing step crashes. There should be a way to see errors for failed configs without inducing a crash.

a bunch of things break when the local repository root has a space in it.

For configuration of source-interface for dns, ntp, logging, snmp, tacacs servers, we need to record any setting of source-interface used to communicate with such servers.

Implement infrastructure for creating, manipulating, and answering an analysis, which is a collection of questions.

router bgp
neighbor next-hop-self

If a differential question is asked when the delta testrig is not set, we get the following error message. A more direct message pointing out the problem will be more helpful.

org.batfish.common.BatfishException: Batfish job failed
at org.batfish.main.Driver$1.run(Driver.java:366)
Caused by: java.lang.NullPointerException
at java.nio.file.Files.provider(Files.java:97)
at java.nio.file.Files.exists(Files.java:2385)
at org.batfish.main.Batfish.checkBaseDirExists(Batfish.java:662)
at org.batfish.main.Batfish.environmentExists(Batfish.java:1211)
at org.batfish.main.Batfish.initQuestionEnvironment(Batfish.java:1969)
at org.batfish.main.Batfish.initQuestionEnvironments(Batfish.java:1988)
at org.batfish.main.Batfish.answer(Batfish.java:425)
at org.batfish.main.Batfish.run(Batfish.java:3340)
at org.batfish.main.Driver$1.run(Driver.java:333)

Right now, a pain point for Batfish setup is installing z3. It would be better if we could simply produce an uber-jar that contains native libraries for major platforms and let users just use that jar.

This would reduce developer requirements for main batfish to just JDK.

Currently, vlan interfaces are considered to be up independently of interfaces that may use them. However, any vlan interface not used by an active switchport interface via trunking or access mode should not be up.

Currently the master grammar files are in projects/batfish/src/main/antlr4/org/batfish/grammar/(name)/(name){Parser,Lexer}.g4.
The imported (subordinate) grammars are all bunched together in projects/batfish/src/main/antlr4/imports/(subordinate_grammar_file).g4.
Ideally each imported grammar would reside in the same directory as the master grammar that imports it.

Current maven build spews warnings about overlapping classes. For each such class, we should choose a representative jar and exclude those classes from all other jars. This should eliminate the warnings.

Currently environment creation can be done one of two ways, neither of which is fully featured:

1. Upload a whole directory as a new environment, which coordinator plops down without doing any validation.
2. Submit the name of a base testrig/environment, and submit node/interface/edge blacklists. Batfish receives these and creates a delta environment off of the base environment for the same testrig. Then it creates blacklist files and writes them to the environment, potentially overwriting any existing blacklists.

In case 1, we are unable to supply base information. In case 2, we are unable to provide all of the things an environment might contain, e.g. changed configs, dumped routes, dumped bgp advertisements.

We should unify these 2 into a two step process with 1 environment-related API call to coordinator, and 1 new batfish command.
The coordinator API call should accept a base testrig and an environment zip. The base environment would be optional. Blacklists, etc. should be packaged in the zip before coordinator receives them.
We should add a new batfish command to validate and finalize an environment. Clients should call this command after initializing a new environment via coordinator before using the new environment.
Environment validation will initially consist of:

• checking blacklists for references to non-existent nodes/interfaces/edges
• processing provided routing and bgp tables
• processing provided changed configs
• providing pass/fail answer (strictness wrt parsing should be configurable)

Support recording of Arista Cisco-like switchport line within interface definitions.

The stored value of the name of the next-hop interface of a static route is not canonicalized. This causes a crash in data plane computation when the actual interface cannot be found due to the name mismatch.