Giter VIP home page Giter VIP logo

tsnsched's Introduction

TSNsched

TSNsched uses the Z3 theorem solver to generate traffic schedules for Time Sensitive Networking (TSN), and it is licensed under the GNU GPL version 3 or later.

This repository is a result of research conducted at fortiss and the Federal University of Paraíba (UFPB) to develop a Time-Aware Shaper for TSN systems. The theoretic basis of this implementation has been published in a paper called TSNsched: Automated Schedule Generation for Time Sensitive Networking. The general idea of TSNsched was also discussed in depth a Master's dissertation of the same title.

Table of Contents

Quickstart Guide

On Linux run the shell script as root to set up all necessary dependencies. The script will also generate an example schedule.

sudo ./install-dependencies.sh

If you already have all dependencies installed, run the following commands for an example schedule.

cd Script/
./generateSchedule.sh example.java

example.java describes a network topology (10 switches, 50 devices) with one small flow (4 subscribers and 3 switches in the path tree):

P -> SW1 -> SW2 -> Sub1
     |       |
     |       V
     |       Sub2
     V        
     SW3 -> Sub3
     |
     V
    Sub4

TSNsched writes the generated TSN schedule to the output directory.

The total execution time, average latency and average jitter of the topology can be found at the end of the output/example.java.out file.

You can find the network topologies described in the submitted paper in TestCases.

You can also generate other topologies using our flow generator. Generating topologies explains how to use this generator.

Requirements

  • Java Version 1.8.0_181
  • Z3 package version 4.8.0.0
  • GNU bash version 4.4.19(1)-release (x86_64-pc-linux-gnu)
  • Eclipse IDE 4.8.0

The Input

Internally, TSNsched is capable of perceiving the network as a set of devices, switches and data streams (commonly referred as flows) and these elements can be represented with a JSON object. Example topologies containing possible configurations for TSNsched can be found in the scripts folder.

The description of the fields of the json elements describing the network topology (input) is shown bellow:

Device

  • name: name of the device;
  • defaultPacketSize: (optional) default size of every frame sent by this device. Can be overridden by the packetSize variable on the flow object;
  • defaultPacketPeriodicity: (optional) default interval between frame sendings expressed as time. Can be overridden by the packetPeriodicity variable in the flow object;
  • defaultHardConstraintTime: (optional) default maximum latency of every frame sent by this device expressed as time. Can be overridden by the hardConstraintTime variable in the flow object;
  • defaultFirstSendingTime: (optional) default moment in time in which the first packet of the device is sent expressed as time. Can be overridden by the firstSendingTime variable in the flow object. If the device is the source of multiple flows and the flows do not override the first sending time with different values, or if the transmission of its first packets overlap, this variable will be ignored and a new value for it will be given as output for both flows;

[INPUT]

TSNSwitch

  • name: name of the switch;
  • defaultTimeToTravel: (optional) default time taken to travel between the port of the switch and the egress queue of the node it connects to. Can be overridden by the timeToTravel variable of the object. Expressed as time;
  • defaultPortSpeed: (optional) default transmission speed of the ports of the switch. Can be overridden by the portSpeed variable of the Port object. Expressed as size per time;
  • defaultGuardBandSize: (optional) default size of the guard bands in the port. Can be overridden by the guardBandSize variable in the Port object. Expressed as size;
  • defaultScheduleType: (optional) default schedule type of all the ports. Can be overridden by the scheduleTypeVariable in the Port object;
  • defaultSlotArrangementMode: (optional) default slot arrangement mode of all the ports of the switch. It can be overridden by the slotArrangementMode variable on the Port;
  • port* [connectsTo]
    • name: name of the port element;
    • connectsTo: name of the node it connects to;
    • timeToTravel: (optional) time taken to travel between the port of the switch and the port of the node it connects to. Overrides the default of the switch. Expressed as time;
    • guardBandSize: (optional) size of the guard bands in the port. Overrides the default guard band size of the switch. Expressed as size;
    • maximumSlotDuration: (optional) maximum size of all the transmission windows (space of time between the egress gate opening and closing on a port) of the port. Overrides the default maximum slot duration of the switch. Expressed as time;
    • cycleStart: (optional) first cycle start of the cycle of the port;

Flow

  • name: name of the flow
  • fixedPriority: (optional) a boolean variable which, if set to true, will force the flow to have the same priority over all its hops. If set to false, the priority of the flow can change from hop to hop, considering that the network has support for this feature (based on the 802.1Qci standard);
  • priorityValue: (optional) an integer variable where its value ranges from -1 to 7. If it is -1, this variable is given as output. If it is any other value, as long as the fixedPriority variable is set to true, the scheduler will force the flow to be scheduled on the priority number specified by this variable;
  • firstSendingTime: (optional) moment in time in which the first frame sent in this stream of frames is sent. Overrides the defaultFirstSendingTime of the Device object. If the source device is also the source of another flow, and the flows do not override the first sending time with different values, or if the transmission of its first packets overlap, the input value for this variable will be ignored and a new value for it will be given as output. Expressed as time;
  • packetPeriodicity: (optional) interval between frame sendings expressed as time. Overrides the defaltPacketPeriodicity variable in the Device object;
  • hardConstraintTime: maximum latency of the packets of that flow;
  • maximumJitter: (optional) maximum variation of the latency of the packets of that flow;
  • sourceDevice: name of the source device
  • endDevice* [name] - List of names of end devices;
    • name: name of end device;
  • hop* [nextNodeName]
    • currentNodeName: name of the node from where the packet departs on the hop;
    • nextNodeName: next node in the frame’s path;

[OUTPUT]

Flows

  • name: name of the flow
  • averageLatency: average latency of all the packets of the flow
  • jitter: average variation of all the packets of the flow
  • firstSendingTime: moment in time where the first packet of the flow is sent
  • priority: (if fixed priority is true in the input) priority of the packets of the flow
  • hops* (used to specify the priority of each hop, if the flow has no fixed priority)
    • currentNodeName: name of the current node in the path
    • nextNodeName: name of the next node in the path
    • priority: priority of the stream in the egress port of the current node

Switch

  • name: name of the switch
  • ports* [name] - cycleDuration: duration of the cycle in the port - firstCycleStart: moment in time where the first cycle starts in the port
    - prioritySlotData* - priority: priority number of the transmission window - slotsData* - slotDuration: duration of the transmission window
    - slotStart: moment in time where the transmission window starts

Command Line Usage

Primarily, TSNsched is compiled as a executable .jar file. This file can be found in the libs folder under the name of TSNsched.jar.

To execute TSNsched, please run the following command:

 java -jar TSNsched.jar INPUT_FILE_NAME

To use the provided sample file as input, replace INPUT_FILE_NAME with input.json.

By default, TSNsched will generate a file titled output.json, which contains the generated schedule organized into json elements. These elements have the necessary information to deploy the schedules in the topology hardware.

TSNsched supports multiple parameters command line execution. The parameters and their description can be seen below:

  • exportModel: Exports the SMT-solver model generated by TSNsched;
  • generateSimulationFiles: Exports files used as input for simulating the generated schedule in omnet++;
  • serializeNetwork: Serializes the network configuration for future use. The schedule can be loaded by using the “-loadNetwork” parameter;
  • loadNetwork: Used to load the serialized topology into the scheduler;
  • enableConsoleOutput: Enables console output for debugging and visual feedback of the tool;
  • enableLoggerFile: Generates a readable file containing the information of the schedule;
  • disableJSONOutput: Used to stop the tool from generating the JSON output; comment: <> ( - useIncremental: When used, enable the incremental scheduling approach to be used.)

Alternatively, this project accompanies a script to execute the scheduler with the necessary configuration for exporting human readable output, and the files used in this approach are stored in the folder Script in this repository. They can be downloaded and used separately.

If the user is not interested in building his own network, we also make available a topology generator, discussed later in this file. The output of this generator is already in the format accepted by the execution script. Samples generated by this tool can be found in the folder "TestCase" and are indentified by the .java extension. We discourage the usage of the input for TSNsched as java files, as it is gradually becoming deprecated in favor of the json input.

This file must be placed inside the folder "Script". The name of the file does not matter, as it will be an input on the command line.

For the script usage, a script was developed in order to compile the Java file containing the network topology, execute it and handle the input and output files.

Once the input file is placed in the same folder of the script, the user must execute the script giving the java file containing the topology as an argument. Given that the name of the file is "example.json", the command will look like the following:

./generateSchedule.sh example.json

For practical reasons, the given file will be duplicated, renamed, parsed by the script in order to adapt the code. Then, the files will be compiled and executed with references to the Z3 and TSNsched libraries, placed on the subfolder "libs". The 2 output files will be generated and placed on the folder "output" under the the same name of the argument given in the execution of the switch, but now with the extra extensions .out and .log.

Complimentary output

The output of this process can be found in the "output" subfolder. If the network topology was specified on a file called "topology.json", then the user should be able to find two new files in the output folder called "topology.java.out" and "topology.java.log", if there was no problem executing the script.

The files with the extension .out contain the printed model generated by Z3 with the extra output created by the user (optional).

The files with the extension .log contain the information about the topology, as well as the Z3 values generated for the properties of the network. These files will be divided in a list of switches and a list of flows.

The list of switches contains individual information about each switch (such as transmission time and time to travel) and its ports (virtual index, first cycle start and duration for debugging purposes. Mostly redundant).

The list of flows contains individual information about each flow. Here, the user can check the flow fragments to retrieve the values of the priority of the flow on a certain switch, the slot start and duration of that flow, and the arrival, departure and scheduled times of each of the packets that goes through the switch covered by this fragment of the flow.

Currently, the scheduler is building the schedule for 5 packets sent by each flow, which can be configured for different settings. Due to this, the user might indentify a pattern on the log files. A index of the packet between parenthisis can be seen followed by the departure, arrival and scheduled times. After this, a dashed line will be printed, separating it from the information about the next packet of the same flow.

Using as a Library

This tool also can be imported as a library allowing the user to aggregate TSNsched functionalities to other projects. With this, not only the topology can be handled as the developer wishes, the values generated by the scheduler will be stored in the cycle and flow objects in the program, allowing users to manipulate data without waiting for an output of a program external to their projects.

Setting up the network

After adding the TSNsched and Z3 packages to the Java build path, one only needs to import the classes in order to be able to make use of it:

import schedule_generator.*;

With this, components of the network can be created:

// Creating a device
Device dev = new Device(float packetPeriodicity,  //  Periodicity of the packet
                        float firstT1Time,        //  First sending time of the device (Check toZ3 method on the device object to see if it is being used)
                        float hardConstraint,     //  Maximum latency tolerated by this device (Hard constraint)
                        float packetSize);        //  Size of packet sent by the device
                
// Creating a switch
TSNSwitch switch = new TSNSwitch(String name,   	 // Identifier of the switch
				       float timeToTravel,     // Time taken to travel on the medium connected to this switch
			         float portSpeed,        // Transmission speed of the port
			         float gbSize)           // Size of the guardband used in the port in time units
				 
// Creating a cycle
Cycle cycle = new Cycle(float maximumSlotDuration);   // Maximum duration of a time window of the cycle

// With the cycle, create ports. 
// First parameter is the device that is being connected to the switch, second is the cycle of the port.
switch.createPort(Device deviceA, Cycle cycle1); 
switch.createPort(Switch switchB, Cycle cycle2); 

// Creating a unicast flow:
Flow flow = new Flow(Flow.UNICAST);

// Setting start device, path and end device of a unicast flow
flow.setStartDevice(Device devA);
flow.addToPath(Switch switchA);
flow.addToPath(Switch switchB);
flow.setEndDevice(Device devB);
	

// Creating a publish subscribe flow:
Flow flow = new Flow(Flow.PUBLISH_SUBSCRIBE);
flow.setStartDevice(Device devA);
// Since now the path can be a tree, the source must also be informed
flow.addToPath(Device devA, Switch switchA); // Adding path from devA to switchA
flow.addToPath(Switch switchA, Switch switchB);
flow.addToPath(Switch switchB, Switch switchC);
flow.addToPath(Switch switchB, Switch switchD);
flow.addToPath(Switch switchC, Device devB);
flow.addToPath(Switch switchD, Device devC);


// Creating and populating a network (Giving switches and flows to it):
Network net = new Network(float jitterUpperBoundRange); // Creating a network giving the maximum average jitter allowed per flow
net.addDevice(Device devA);
net.addDevice(Device devB);
net.addDevice(Device devC);
net.addSwitch(Switch switchA); 
net.addSwitch(Switch switchB); 
net.addSwitch(Switch switchC); 
net.addFlow(Flow flowA); 
net.addFlow(Flow flowB); 

Executing the program

The user must add both Z3 and TSNsched packages to the classpath of the project. These two files can be found in the "libs" folder of this repository.

Most of the sofisticated IDEs can do this just by adding external libraries as JAR files on the configuration of the projects.

If compiling the project in the command line, do not forget to add the libraries manually or set them in the Java PATH environment variable.

After setting up the network, the user must now call the method for generating a schedule:

// Generating a schedule:
ScheduleGenerator scheduleGenerator = new ScheduleGenerator();
scheduleGenerator.generateSchedule(Network net); // The network is the input for the schedule generation

The output

By default, TSNsched will generate a .json output file containing the information about the flows, its individual packets and the gate control list information of the switches. The output is structured as follows:

[Flows]

  • name: name of the flow
  • averageLatency: average latency of all the packets of the flow
  • jitter: average variation of all the packets of the flow
  • firstSendingTime: moment in time where the first packet of the flow is sent
  • priority: (if fixed priority is true in the input) priority of the packets of the flow
  • hops* (used to specify the priority of each hop, if the flow has no fixed priority)
    • currentNodeName: name of the current node in the path
    • nextNodeName: name of the next node in the path
    • priority: priority of the stream in the egress port of the current node

[Switch]

  • name: name of the switch
  • ports* [name] - cycleDuration: duration of the cycle in the port - firstCycleStart: moment in time where the first cycle starts in the port
    - prioritySlotData* - priority: priority number of the transmission window - slotsData* - slotDuration: duration of the transmission window
    - slotStart: moment in time where the transmission window starts

If the logging functionality is enabled on input, a "log.txt" file must be generated within the project folder. This file contains the information about the topology, as well as the Z3 values generated for the properties of the network (such as cycle start and duration, priorities and packet times).

If importing TSNsched in a library in your project, it is possible to return numeric data to the user accessing the topology object as follows:

// Retreiving the departure time from a packet in a publish subscribe flow 
flow.getDepartureTime(Device targetDevice,     // Destination of the packet. One of the subscribers 
                      int switchNum,           // Index of the switch in the path
                      int packetNum);          // Index of the packet in the sequence

// Retreiving the arrival time from a packet in a publish subscribe flow 
flow.getArrivalTime(Device targetDevice,       // Destination of the packet. One of the subscribers 
                    int switchNum,             // Index of the switch in the path
                    int packetNum);            // Index of the packet in the sequence

// Retreiving the scheduled time from a packet in a publish subscribe flow 
flow.getScheduledTime(Device targetDevice,       // Destination of the packet. One of the subscribers 
                      int switchNum,             // Number of the switch in the path
                      int packetNum);            // Index of the packet in the sequence

// Retrieving the average latency and average jitter of a flow, respectively:
flow.getAverageLatency();
flow.getAverageJitter();

// Retrieving the cycle duration and cycle start of a flow:
cycle.getCycleDuration();
cycle.getCycleStart();

// Retrieving the index of priorities used:
cycle.getSlotsUsed();

// Retrieving the priority slot duration and priority slot start:
cycle.getSlotStart(int prt);        // Index of a priority
cycle.getSlotDuration(int prt);     // Index of a priority

Generating topologies

Check the full description on how to generate arbitrary TSNsched input topologies here.

Simulation Parser

TSNsched offers means of validation through the translation of the values contained in the Network object created in the scheduling generation process. Using the Network object, the parser creates the necessary files for the NeSTiNg simulation model that uses the OMNeT++ and INET Framework.

Generating Simulation Files

The generation of simulation files is fully automated. At the end of the default TSNsched scheduling generation, the creation of the necessary files for the simulation begins, being these files:

  • Network Description (.ned)
  • Initialization (.ini)
  • Port Scheduling (.xml)
  • Routing (.xml)
  • Traffic Generator (.xml)

All of these files can be found in the folder named nestsched.

Running the Simulation

To run the simulation and validade the generated scheduling, it's necessary to have NeSTiNg, a simulation model for Time Sensitive Networking that currently uses OMNeT++ version 5.5.1 and INET version 4.1.2.

With the tools properly installed, it is now necessary to make some changes to the source code of NeSTiNg and INET so that it is possible to generate the analysis signals that we need and calculate the latency of communication between devices.

The first change to be made is in the INET Framework, more precisely in the Ethernet.h file. In this file, it is necessary to change the value of the constant that defines INTERFRAME_GAP_BITS from 96 to 0. The Interframe Gap is the minimum pause necessary for the receiver to be able to make clock recoveries, allowing it to prepare itself for receiving the next packet. This change is necessary because TSNsched does not take Interframe Gap into account when generating the network schedule.

Next we need to change the QueuingFrames.h file in the NeSTiNg. In this file, the last line of the matrix standardTrafficClassMapping has to follow the ascending order from 0 to 7, so that the packets can be routed correctly according to the TSNsched configuration.

And finally we need to change the VlanEtherTafGenSched.cc and VlanEtherTafGenSched.h files. In these files, we need to change three things: (i) we need to track the number of packets sent, so the maximum number of packets determined by the generated schedule are followed. (ii) we need to change the packet name, putting the flowId in the beginning of the name, so we can easily track that information later. (iii) and we need to create and initialize the flowId signals and generate them when a packet is received.

With all these changes made, now you just have to generate the network scheduling with TSNsched, copy the nestsched folder into the NeSTiNg examples folder and run the simulation in the OMNeT++.

Overview of Classes

Check the full description of TSNsched classes here.

Frequently Asked Questions

Check the full TSNsched's FAQ here.

tsnsched's People

Contributors

acassimiro avatar github-throwaway avatar piumoreira avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

tsnsched's Issues

Add configuration options at port level.

Enable configuration for the following properties at port level:

  • timeToTravel float (input)
  • portSpeed float (input)
  • gbSize float (input)
  • name string

Enable user to control port name

no output

Hi,

I have all dependencies installed successfully by running "sudo ./install-dependencies.sh".

But, after running the following commands,
cd ./Script
./generateSchedule.sh example.java

It display the following text:
:./libs/com.microsoft.z3.jar:./libs/gson-2.8.6.jar:./libs/java-json.jar:./libs/TSNsched.jar
mv: cannot stat 'log.txt': No such file or directory
Ending execution

and Script/output/example.java.out is empty.

Can you help me?

There is a bug after modifying M1 chip

Hello, I installed all the dependencies according to the tutorial, but there are bugs similar to others.
Then I found that the bug occurred after the M1 compatibility was modified.
So other users can go back to the version before M1 modification, and then they can run normally.

i am chinese
您好,我按教程安装了所有的依赖,但是出现了跟其他人类似的bug。
然后我发现,应该是在修改M1兼容性以后,出现了bug。
所以其他用户可以回退到M1修改之前的那个版本,就可以正常运行了。

Two flows with different cycle merge into the same link

Hi,
Thanks for your code. I am running an example on the project.
My example has two flows as follows:

Flow flow0 = new Flow(Flow.UNICAST);
flow0.setStartDevice(dev4);
flow0.addToPath(switch0);
flow0.addToPath(switch2);
flow0.setEndDevice(dev10);
flow0.setFlowSendingPeriodicity(2000);


Flow flow1 = new Flow(Flow.UNICAST);
flow1.setStartDevice(dev5);
flow1.addToPath(switch1);
flow1.addToPath(switch2);
flow1.setEndDevice(dev10);
flow1.setFlowSendingPeriodicity(3000);

The topology is:

Dev4 -> SW0 -> SW2 -> Dev10
                ^
                |
Dev5 -> Sw1 -> 

And the result is:

>>>> INFORMATION OF SWITCH: switch0 <<<<
    Port list - 
        => Port name:       switch0Port1
        Connects to:     switch2
        Cycle start:    0.0
        Cycle duration: 2000.0
        Fragments:       flow1Fragment1, 
        Priority number: 7
          Index 0 Slot start:      1949.088
          Index 0 Slot duration:   8.004
        ------------------------


>>>> INFORMATION OF SWITCH: switch1 <<<<
    Port list - 
        => Port name:       switch1Port1
        Connects to:     switch2
        Cycle start:    0.0
        Cycle duration: 3000.0
        Fragments:       flow2Fragment1, 
        Priority number: 7
          Index 0 Slot start:      2940.996
          Index 0 Slot duration:   8.0
        ------------------------


>>>> INFORMATION OF SWITCH: switch2 <<<<
    Port list - 
        => Port name:       switch2Port9
        Connects to:     dev10
        Cycle start:    0.0
        Cycle duration: 1000.0
        Fragments:       flow1Fragment2, flow2Fragment2, 
        Priority number: 7
          Index 0 Slot start:      949.992
          Index 0 Slot duration:   50.0
        ------------------------

I notice that Cycle duration of Switch2 is 1ms. It means that there is a slot in 1ms. But cycle of flow0 is 2ms, and cycle of flow1 is 3ms. There will has two slot without packets within every 6ms.

I think it is better that the Cycle duration of Switch2 shall be 6ms (i.e. Least Common Multiple of two flow). In each cycle, there are four time slots .
dbe31be1-9365-4c04-b278-f14dcfd5f216

Which part of the code do I need to modify?
Thanks

Simplify language of README

The documentation is currently very extensive but also hard to read. I propose the following tools to improve it:

Hemingway makes your writing bold and clear. It's like a spellchecker, but for style. It makes sure that your reader will focus on your message, not your prose.

LanguageTool is an Open Source proofreading software for English, French, German, Polish, Russian, and more than 20 other languages. It finds many errors that a simple spell checker cannot detect.

Compose clear, mistake-free writing that makes the right impression with Grammarly’s writing assistant.

Failed to run the example

image
Hello, I am using Ubuntu18.04, JAVA-8, Z3 4.8.10 or Z34.9.1 I have also used, but when I run the example, there is always an error, the error is as shown below, do you have a better suggestion

Java version issue when trying the example

Hi!

Apologies for the beginner question. I am trying to run the example use case under the Script directory. The ./install-dependencies.sh script finished without issues until the example. At first, I unfortunately had this error when running the example script with Java 8 or 11:

$ ./generateSchedule.sh example.java
rm: cannot remove 'output/*': No such file or directory
:./libs/com.microsoft.z3.jar:./libs/gson-2.8.6.jar:./libs/java-json.jar:./libs/TSNsched.jar
UseCase.java:1: error: cannot access JSONParser
import com.tsnsched.core.interface_manager.JSONParser;
                                          ^
  bad class file: ./libs/TSNsched.jar(/com/tsnsched/core/interface_manager/JSONParser.class)
    class file has wrong version 61.0, should be 55.0
    Please remove or make sure it appears in the correct subdirectory of the classpath.
[...]

It seemed like the .jar lib files were complied using Java 17 (version 61.0 shown above) while version 11 was expected (version 55.0). The same happens when trying with Java 8, except 55.0 is replaced by 52.0.

I then tried to install Java 17 and run the script with it, but I get a different error:

$ ./generateSchedule.sh example.java                                        
rm: cannot remove 'output/*': No such file or directory
:./libs/com.microsoft.z3.jar:./libs/gson-2.8.6.jar:./libs/java-json.jar:./libs/TSNsched.jar
Exception in thread "main" java.lang.NoSuchMethodError: 'void com.microsoft.z3.Solver.add(com.microsoft.z3.Expr[])'
        at com.tsnsched.core.components.Port.setUpCycle(Port.java:1377)
        at com.tsnsched.core.nodes.TSNSwitch.setUpCycleSize(TSNSwitch.java:418)
        at com.tsnsched.core.schedule_generator.ScheduleGenerator.configureNetwork(ScheduleGenerator.java:159)
        at com.tsnsched.core.schedule_generator.ScheduleGenerator.generateSchedule(ScheduleGenerator.java:257)
        at UseCase.runTestCase(UseCase.java:453)
        at GenerateScheduleJavaInput.main(GenerateScheduleJavaInput.java:15)
mv: cannot move 'output.txt' to 'output/example.java.out': No such file or directory
mv: cannot stat 'log.txt': No such file or directory
mv: cannot stat 'output.json': No such file or directory
Ending execution

Is there an obvious mistake? I see that the installation script installs Java 8, but I haven't figured out what could be wrong.
Thank you for your help!

Enable ports with the same name in different switches

TSNsched uses the name of the port to specify some variables to the SMT solver. The solver does not care about the structure of the ports or switches, and if two ports have the same name, there will be multiple variables representing different elements of the network to the solver with the same name as well.

Until this get fixed, please, use unique name to all ports, or do not specify the name of the port at all.

Incompatibilities with latest Z3 releases

It seems that TSNSched has some problems with the latest Z3 releases (4.10 and above)

:./libs/com.microsoft.z3.jar:./libs/gson-2.8.6.jar:./libs/java-json.jar:./libs/TSNsched.jar
java: symbol lookup error: /usr/lib/libz3java.so: undefined symbol: Z3_toggle_warning_messages

Installing Z3 v4.9.1 seems to solve the problem.

Clearly, I also had also to revert the changes for the m1 chip as described in #24

when set two flows with different periods,cant get the reult,which log the "The specified constraints MIGHT NOT be satisfiable"

here is the test code,which overwrites the SmallScenario.java
the network is 4 devices and 6 switches and 2 flows,when flow1 send at 2ms, flow2 send at 2ms,the result is ok
when flow1 send at 2ms, flow2 send at 3ms,the result is NOT be satisfiable.
but i dont know why,becasue the z3 just give the final result,but which constraint is unknown.

package com.tsnsched.generated_scenarios;
import java.util.;
import java.io.
;
import com.tsnsched.core.nodes.;
import com.tsnsched.core.components.Cycle;
import com.tsnsched.core.components.Flow;
import com.tsnsched.core.components.PathNode;
import com.tsnsched.core.network.
;
import com.tsnsched.core.schedule_generator.*;
import com.tsnsched.core.components.Port;

public class SmallScenario {
public void runTestCase(){

	/*
	 * GENERATING DEVICES
	 */
	Device ccu = new Device(1000, 0, 1000, 1500);
	ccu.setName("ccu");
	
	Device ccu2 = new Device(1000, 0, 1000, 1500);
	ccu2.setName("ccu2");
	
	Device tcu = new Device(1000, 0, 1000, 1500);
	tcu.setName("tcu");
	
	Device bcu = new Device(1000, 0, 1000, 1500);
	bcu.setName("bcu");

	/*
	 * GENERATING SWITCHES
	 */
	TSNSwitch switch0 = new TSNSwitch("switch0", 100, 1, 125, 1, 400, 3000);
	TSNSwitch switch1 = new TSNSwitch("switch1", 100, 1, 125, 1, 400, 3000);
	TSNSwitch switch2 = new TSNSwitch("switch2", 100, 1, 125, 1, 400, 3000);
	TSNSwitch switch3 = new TSNSwitch("switch3", 100, 1, 125, 1, 400, 3000);
	TSNSwitch switch4 = new TSNSwitch("switch4", 100, 1, 125, 1, 400, 3000);
	TSNSwitch switch5 = new TSNSwitch("switch5", 100, 1, 125, 1, 400, 3000);
	
	/*
	 * GENERATING SWITCH CONNECTION PORTS
	 */
	Cycle cycle0 = new Cycle(50);
	Port port0 = switch0.createPort(switch1, cycle0);//switch0 use x11 to connect switch1 
	port0.setPortNum(11);
	port0.setName(port0.getName()+ "Port"+ "11");
	
	Cycle cycle1 = new Cycle(50);
	Port port1 = switch1.createPort(switch0, cycle1);
	port1.setPortNum(4);
	port1.setName(port1.getName()+ "Port"+ "4");
	

	Cycle cycle2 = new Cycle(50);
	Port port2 = switch1.createPort(switch2, cycle2);	
	port2.setPortNum(1);
	port2.setName(port2.getName()+ "Port"+ "1");
	
	Cycle cycle3 = new Cycle(50);
	Port port3 = switch2.createPort(switch1, cycle3);
	port3.setPortNum(2);
	port3.setName(port3.getName()+ "Port"+ "2");
	
	Cycle cycle4 = new Cycle(50);
	Port port4 = switch2.createPort(switch3, cycle4);
	port4.setPortNum(1);
	port4.setName(port4.getName()+ "Port"+ "1");
	
	Cycle cycle5 = new Cycle(50);
	Port port5 = switch3.createPort(switch2, cycle5);
	port5.setPortNum(2);
	port5.setName(port5.getName()+ "Port"+ "2");
	
	Cycle cycle6 = new Cycle(50);
	Port port6 = switch3.createPort(switch4, cycle6);
	port6.setPortNum(1);
	port6.setName(port6.getName()+ "Port"+ "1");
	
	Cycle cycle7 = new Cycle(50);
	Port port7 = switch4.createPort(switch3, cycle7);
	port7.setPortNum(2);
	port7.setName(port7.getName()+ "Port"+ "2");
	
	Cycle cycle8 = new Cycle(50);
	Port port8 = switch4.createPort(switch5, cycle8);
	port8.setPortNum(4);
	port8.setName(port8.getName()+ "Port"+ "4");
	
	Cycle cycle9 = new Cycle(50);
	Port port9 = switch5.createPort(switch4, cycle9);	
	port9.setPortNum(11);
	port9.setName(port9.getName()+ "Port"+ "11");


	
	
	/*
	 * LINKING SWITCHES TO DEVICES
	 */
	Cycle cycle10 = new Cycle(50);
	Port port10 = switch0.createPort(ccu, cycle10);
	port10.setPortNum(9);
	port10.setName(port10.getName()+ "Port"+ "9");
	
	Cycle cycle11 = new Cycle(50);
	Port port11 = switch5.createPort(tcu, cycle11);
	port11.setPortNum(9);
	port11.setName(port11.getName()+ "Port"+ "9");
	
	Cycle cycle12 = new Cycle(50);
	Port port12 = switch2.createPort(ccu2, cycle12);
	port12.setPortNum(9);
	port12.setName(port12.getName()+ "Port"+ "9");
	
	
	Cycle cycle13 = new Cycle(50);
	Port port13 = switch4.createPort(bcu, cycle13);
	port13.setPortNum(9);
	port13.setName(port13.getName()+ "Port"+ "9");
	
	/*
	 * GENERATING FLOWS
	 */
	LinkedList<PathNode> nodeList;

	Flow flow1 = new Flow(1,0,2000);
	//Flow flow1 = new Flow(Flow.UNICAST);
	flow1.setStartDevice(ccu);
	flow1.addToPath(ccu,switch0);
	flow1.addToPath(switch0,switch1);
	flow1.addToPath(switch1,switch2);
	flow1.addToPath(switch2,ccu2);
	flow1.addToPath(switch2,switch3);
	flow1.addToPath(switch3,switch4);
	flow1.addToPath(switch4,bcu);
	flow1.addToPath(switch4,switch5);
	flow1.addToPath(switch5,tcu);
	
	flow1.setPriorityValue(5);
	flow1.setFixedPriority(true);
	//flow1.setPacketSize(1500);
	//flow1.setFlowMaximumLatency(1000);
	flow1.setFlowSendingPeriodicity(2000);
	
	
	Flow flow2 = new Flow(1,0,2000);
	//Flow flow2 = new Flow(Flow.UNICAST);
	flow2.setStartDevice(ccu);
	flow2.addToPath(ccu,switch0);
	flow2.addToPath(switch0,switch1);
	flow2.addToPath(switch1,switch2);
	flow2.addToPath(switch2,ccu2);
	flow2.addToPath(switch2,switch3);
	flow2.addToPath(switch3,switch4);
	flow2.addToPath(switch4,bcu);
	flow2.addToPath(switch4,switch5);
	flow2.addToPath(switch5,tcu);
	flow2.setPriorityValue(5);
	flow2.setFixedPriority(true);	
	//flow2.setPacketSize(1500);
	flow2.setFlowSendingPeriodicity(2000);//when set it to 3ms,the result is not satisfied
	//flow2.setFlowMaximumLatency(1000);
	
	/*
	 * GENERATING THE NETWORK
	 */
	Network net = new Network(25);
	net.addDevice(ccu);
	net.addDevice(ccu2);
	net.addDevice(tcu);
	net.addDevice(bcu);

	net.addSwitch(switch0);
	net.addSwitch(switch1);
	net.addSwitch(switch2);
	net.addSwitch(switch3);
	net.addSwitch(switch4);
	net.addSwitch(switch5);
	
	net.addFlow(flow1);
	net.addFlow(flow2);
	
	ScheduleGenerator scheduleGenerator = new ScheduleGenerator(false);
	scheduleGenerator.generateSchedule(net);


}

}

Questions about Implementing TSNsched

Hello, I've been working on implementing TSNsched recently, and there are a few things I don't quite understand.

First: Is this the correct way to set up HyperCycle? Which parts need modification?
01

Second: If I want to set it to milliseconds, what code should I add?

Third: Regarding the Total number of scheduled packets in the .out file, I checked the paper, but it doesn't seem to mention this part, so I don't understand what it means.
02

Sorry for the numerous questions and thank you for your response.

What went wrong?

Hello, thanks for your code, but I have some problems. I use version Ubuntu18.04, z3-4.8.13, but running examples always report this error。Please advise?

kaiguoguo@kaiguoguo-virtual-machine:~/Desktop/TSNsched/Script$ ./generateSchedule.sh example.java
:./libs/com.microsoft.z3.jar:./libs/gson-2.8.6.jar:./libs/java-json.jar:./libs/TSNsched.jar
Exception in thread "main" java.lang.NoSuchMethodError: 'void com.microsoft.z3.Solver.add(com.microsoft.z3.Expr[])'
at com.tsnsched.core.components.Port.setUpCycle(Port.java:1377)
at com.tsnsched.core.nodes.TSNSwitch.setUpCycleSize(TSNSwitch.java:418)
at com.tsnsched.core.schedule_generator.ScheduleGenerator.configureNetwork(ScheduleGenerator.java:159)
at com.tsnsched.core.schedule_generator.ScheduleGenerator.generateSchedule(ScheduleGenerator.java:257)
at UseCase.runTestCase(UseCase.java:453)
at GenerateScheduleJavaInput.main(GenerateScheduleJavaInput.java:15)
mv: cannot stat 'log.txt': No such file or directory
mv: cannot stat 'output.json': No such file or directory
Ending execution

Use Google Java Code Style

Installing the coding style settings in Eclipse

Google Java Code Style

Download the eclipse-java-google-style.xml. Under Window/Preferences select Java/Code Style/Formatter. Import the settings file by selecting Import.

Configure save actions to automatically format your code when saving

In the Preferences menu, select Java -> Editor -> Save Actions.

Select Perform the selected actions on save.
Select Format source code.
Click the Formatter” link and ensure the “GoogleStyle formatter is selected as active.
Click OK.

Now, whenever you make changes to your code, you can use the format source code menu item, or just save the file and formatting will be applied automatically.

How to write code for one device sending two or more TSN flows?

Thanks for your code, Cassimiro.
In the example code, when need a tsn terminal, you define it like this:
"Device dev0 = new Device(1000, 0, 1000, 1625);"

Does it mean that one device can only send one tsn flow? How can it be allowed if we want the device to send two more TSN flows at one port?

Support for multiple streams sourced at a single Device

Hello and thanks for making your source code public!

I am wondering if there is a way to specify multiple streams sourced at a single Talker Device and let solver consider this in the GCL computation? The Device class interface seems to take as input only a single set of packetPeriodicity + XConstraintTime + packetSize arguments.

Thanks!

Flows with the same source but different periods

Hey there,
thank you for the nice tool!. While trying to use Tsn-Sched, I noticed that when defining flows of the same source node but with different periods, the resulting model is (most of the time) unsatisfiable (even for a simple topology)
For example, suppose we have the topology :


dev1 --> swt1 --> swt2 --> dev2

and 3 flows f1,f2, and f3 (with periods 250, 500, and 1000 respectively) starting from dev1 and ending at dev2.
Even with the simple setup, the solver does not find any solution, so I would like to learn more about the reason behind the unsatisfiability.

So is there anything wrong with my understanding?
Thank you in advance,
George

P.S: The network of the example is defined as follows:

        ////////////////////////////////////////////
        // defining devices && switches
        ////////////////////////////////////////////
        Device dev1 = new Device(250, 0, 10000000, 250);
        Device dev2 = new Device(250, 0, 10000000, 150);
        TSNSwitch swt1 = new TSNSwitch("swt1", 1500, 1, 250, 1, 1, 200000);
        TSNSwitch swt2 = new TSNSwitch("swt2", 1500, 1, 250, 1, 1, 200000);
        ////////////////////////////////////////////
        // defining Cycles && ports
        ////////////////////////////////////////////
        Cycle c1 = new Cycle(200000);
        Cycle c2 = new Cycle(200000);
        Cycle c3 = new Cycle(200000);
        swt1.createPort(swt2, c1);
        swt1.createPort(dev1, c2);
        swt2.createPort(dev2, c3);
        ////////////////////////////////////////////
        // defining Flows
        ////////////////////////////////////////////
        Flow f1 = new Flow("f1", Flow.PUBLISH_SUBSCRIBE);
        f1.setStartDevice(dev1);
        PathNode root1 = f1.getPathTree().getRoot();
        root1.addChild(swt1);
        PathNode pathNode = f1.getPathTree().searchNode("swt1", root1);
        pathNode.addChild(swt2);
        PathNode pathNode2 = f1.getPathTree().searchNode("swt2", root1);
        pathNode2.addChild(dev2);
        f1.setFixedPriority(true);
        f1.setPriorityValue(7);

        Flow f2 = new Flow("f2", Flow.PUBLISH_SUBSCRIBE);
        f2.setStartDevice(dev1);
        PathNode root2 = f2.getPathTree().getRoot();
        root2.addChild(swt1);
        PathNode pathNode3 = f2.getPathTree().searchNode("swt1", root2);
        pathNode3.addChild(swt2);
        PathNode pathNode4 = f2.getPathTree().searchNode("swt2", root2);
        pathNode4.addChild(dev2);
        f2.setFixedPriority(true);
        f2.setPriorityValue(7);

        Flow f3 = new Flow("f3", Flow.PUBLISH_SUBSCRIBE);
        f3.setStartDevice(dev1);
        PathNode root3 = f3.getPathTree().getRoot();
        root3.addChild(swt1);
        PathNode pathNode5 = f3.getPathTree().searchNode("swt1", root3);
        pathNode5.addChild(swt2);
        PathNode pathNode6 = f3.getPathTree().searchNode("swt2", root3);
        pathNode6.addChild(dev2);
        f3.setFixedPriority(true);
        f3.setPriorityValue(7);
        ////////////////////////////////////////////
        // setting periods for the flows
        ////////////////////////////////////////////
        f1.setFlowSendingPeriodicity(250);
        f2.setFlowSendingPeriodicity(500);
        f3.setFlowSendingPeriodicity(1000);
        ////////////////////////////////////////////
        // Defining network
        ////////////////////////////////////////////
        Network net = new Network();
        net.addDevice(dev2);
        net.addDevice(dev1);
        net.addSwitch(swt2);
        net.addSwitch(swt1);
        net.addFlow(f2);
        net.addFlow(f1);
        net.addFlow(f3);
        
        ////////////////////////////////////////////
        // checking the module
        ////////////////////////////////////////////
        ScheduleGenerator scheduleGenerator = new ScheduleGenerator();
        scheduleGenerator.setEnableLoggerFile(true);
        scheduleGenerator.setEnableConsoleOutput(true);
        scheduleGenerator.generateSchedule(net);

How to run the project under OMNet++

Hi there.
Since I've changed my environment to ubuntu18.04, and the OMNet++ version is 5.5.1, and the version of INet is 4.1.2, the INet and nesting package have run well.
However, here is the problem occurred: after I built the TSNSched and run the nestSched.ini file in OMNet++, the IDE could not execute successfully (the new window of simulation didn't pop out) , the error messege showed “Error: Could not load NED sources from '..': Declared package 'nesting.simulations.examples.nestSched' does not match expected package 'nestSched' in file nestSched.ned”
2024-04-23 16-39-30 的屏幕截图
How can I correct this situation? Looking forward to you reply.

Results are different between command line(generateSchedule.sh) and Eclipse java project(use src/*.java files)?

Hello,
i run tests from cmd and Eclipse java project. The test case is same(Script/example.java).But results are a little different between command line(generateSchedule.sh) and Eclipse java project(use src/.java files).
Is there something different from Scripts/libs/
.jar and src/*java files?

Result.tar.gz

I also confused about the value "Fragment slot duration : 28.375(28.5)" , because packetsize is 1625, that means the packet need 13 microsecond to transmit. so i think fragment slot duration 14 microseconds is enough. How to get the value 28?

FLOW LIST:
Flow name: flow1
Start dev. first t1: 0
Start dev. HC: 1000
Start dev. packet periodicity: 1000
Flow type: Multicast
List of leaves: dev15, dev16, dev17, dev18, dev19,
Path to dev15: dev4, switch0(flow1Fragment1), switch4(flow1Fragment2), switch3(flow1Fragment3), dev15,
Path to dev16: dev4, switch0(flow1Fragment1), switch4(flow1Fragment2), switch3(flow1Fragment4), dev16,
Path to dev17: dev4, switch0(flow1Fragment1), switch4(flow1Fragment2), switch3(flow1Fragment5), dev17,
Path to dev18: dev4, switch0(flow1Fragment1), switch4(flow1Fragment2), switch3(flow1Fragment6), dev18,
Path to dev19: dev4, switch0(flow1Fragment1), switch4(flow1Fragment2), switch3(flow1Fragment7), dev19,

Fragment name: flow1Fragment1
    Fragment node: switch0
    Fragment next hop: switch4
    Fragment priority: 0
    Fragment slot start 0: 0.125
    Fragment slot duration 0 : 28.375
    Fragment times-
      (0) Fragment departure time: 0.0
      (0) Fragment arrival time: 1.0
      (0) Fragment scheduled time: 14.0
      ----------------------------
Fragment name: flow1Fragment2
    Fragment node: switch4
    Fragment next hop: switch3
    Fragment priority: 0
    Fragment slot start 0: 0.125
    Fragment slot duration 0 : 28.375
    Fragment times-
      (0) Fragment departure time: 14.0
      (0) Fragment arrival time: 15.0
      (0) Fragment scheduled time: 28.0
      ----------------------------
Fragment name: flow1Fragment3
    Fragment node: switch3
    Fragment next hop: dev15
    Fragment priority: 0
    Fragment slot start 0: 0.125
    Fragment slot duration 0 : 28.375
    Fragment times-
      (0) Fragment departure time: 28.0
      (0) Fragment arrival time: 29.0
      (0) Fragment scheduled time: 42.0
      ----------------------------
Fragment name: flow1Fragment4
    Fragment node: switch3
    Fragment next hop: dev16
    Fragment priority: 0
    Fragment slot start 0: 0.125
    Fragment slot duration 0 : 28.375
    Fragment times-
      (0) Fragment departure time: 28.0
      (0) Fragment arrival time: 29.0
      (0) Fragment scheduled time: 42.0
      ----------------------------
Fragment name: flow1Fragment5
    Fragment node: switch3
    Fragment next hop: dev17
    Fragment priority: 0
    Fragment slot start 0: 0.125
    Fragment slot duration 0 : 28.375
    Fragment times-
      (0) Fragment departure time: 28.0
      (0) Fragment arrival time: 29.0
      (0) Fragment scheduled time: 42.0
      ----------------------------
Fragment name: flow1Fragment6
    Fragment node: switch3
    Fragment next hop: dev18
    Fragment priority: 0
    Fragment slot start 0: 0.125
    Fragment slot duration 0 : 28.375
    Fragment times-
      (0) Fragment departure time: 28.0
      (0) Fragment arrival time: 29.0
      (0) Fragment scheduled time: 42.0
      ----------------------------
Fragment name: flow1Fragment7
    Fragment node: switch3
    Fragment next hop: dev19
    Fragment priority: 0
    Fragment slot start 0: 0.125
    Fragment slot duration 0 : 28.375
    Fragment times-
      (0) Fragment departure time: 28.0
      (0) Fragment arrival time: 29.0
      (0) Fragment scheduled time: 42.0
      ----------------------------

Porting to C++ and merging into INET/OMNeT++

Hi!

My name is Levente Mészáros and I'm one of the core developers of the INET Framework for the OMNeT++ network simulator tool . Recently, INET has been extended with TSN features and I found your TSNsched project very interesting. I would like to port it to C++ and include it in the INET framework project. In the future, I'm planning to add stream redundancy support. Of course, I would give all credits to you in the source files. Do you mind this?

Best wishes,
levy

Priorities of different Fragments

Hello,
Why do different fragments of a stream have different priorities?
What determines this priority?
Wasn't the priority of each stream initially defined by the user?
Another question is, Where is the maximum latency that can be tolerated per flow set? I now want to change the maximum latency of a flow, but I can't find the corresponding code.

The flow type setting problem

I set flow type as UNICAST in my java file, but the output file (.log) show the flow is MULTICAST.
My java file screenshot:
java

The output file screenshot:
log

about cycle start time

Hello,
I used this tool to run a simple test case, and I have a question: the result shows that the cycle start time of the switches are different, which brings inconvenience to the manual verification results later. Can we fix the cycle start time of all switches to be the same? For example, both are 0.

thank you!

INFORMATION OF SWITCH: switch0 <<<<
Port list -
=> Port name: switch0Port1
Connects to: switch2
Cycle start: 0.0
Cycle duration: 1000.0
Fragments: flow1Fragment1,
Slots per prt: 1
Priority number: 5
Index 0 Slot start: 0.5
Index 0 Slot duration: 52.5
------------------------

INFORMATION OF SWITCH: switch2 <<<<
Port list -
=> Port name: switch2Port2
Connects to: switch3
Cycle start: 1.0
Cycle duration: 1000.0
Fragments: flow1Fragment2, flow2Fragment2, flow3Fragment2,
Slots per prt: 1
Priority number: 5
Index 0 Slot start: 0.5
Index 0 Slot duration: 52.5
------------------------

INFORMATION OF SWITCH: switch3 <<<<
Port list -
=> Port name: switch3Port9
Connects to: dev15
Cycle start: 28.0
Cycle duration: 1000.0
Fragments: flow1Fragment3, flow2Fragment3, flow3Fragment3, flow4Fragment2,
Slots per prt: 1
Priority number: 5
Index 0 Slot start: 0.5
Index 0 Slot duration: 52.5
------------------------

INFORMATION OF SWITCH: switch4 <<<<
Port list -
=> Port name: switch4Port2
Connects to: switch2
Cycle start: 0.0
Cycle duration: 1000.0
Fragments: flow2Fragment1, flow3Fragment1,
Slots per prt: 1
Priority number: 5
Index 0 Slot start: 0.5
Index 0 Slot duration: 52.5
------------------------

INFORMATION OF SWITCH: switch5 <<<<
Port list -
=> Port name: switch5Port3
Connects to: switch3
Cycle start: 14.0
Cycle duration: 1000.0
Fragments: flow4Fragment1,
Slots per prt: 1
Priority number: 5
Index 0 Slot start: 0.5
Index 0 Slot duration: 52.5
------------------------

Documentation of new features

TSNsched now enables new features such as JSON and XML input support and more effective scheduling methodologies. This needs to be documented.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.