Giter VIP home page Giter VIP logo

dtf's Introduction

Android Device Testing Framework (dtf)

Build Status

About

The Android Device Testing Framework (dtf) is a data collection and analysis framework to help individuals search for vulnerabilities on mobile devices. dtf is not a:

  • Vulnerability scanner or explitation framework for mobile devices
  • An application assessment tool
  • A "turn your phone into a hacking device" tool

Instead, dtf aims to allow testers to obtain information from their Android device, process this information into databases, and then start searching for vulnerabilities (all without requiring root privileges). These modules help you focus on changes made to AOSP components such as applications, frameworks, system services, as well as lower-level components such as binaries, libraries, and device drivers. In addition, you'll be able to analyze new functionality implemented by the OEMs and other parties to find vulnerabilities.

Key Features

  • 30+ modules for collecting, processing, and interacting with a device
  • Builtin API for interfacing with your target device
  • Python and shell bindings for creating modules
  • Per-project property sub-system and logging/auditing
  • Bundled versions of numerous Android tools (think apktool/smali/dex2jar)

Installing

dtf is offically supported on Ubuntu, particularly versions 14 through 16. At this time there is no support for Windows or OS X.

Manual Prerequisites

The only manual installation requirement for dtf is the Android SDK (dtf relies on the adb utility). It is recommended that you install Android Studio, and add adb to your $PATH.

Installation Script

To install dtf on Ubuntu (or update the framework), run the following commands:

$ curl -sSL thecobraden.com/getdtf > install.sh
$ chmod u+x install.sh
$ ./install.sh

If you're one of those people who doesn't trust the whole curl|bash model, just download the script from the GitHub page.

Managing Installed Content

During installation, dtf will automatically configure itself to pull from the stable feed of core content. It is a good idea to routinely run the following command to ensure dtf remains up-to-date:

$ dtf pm upgrade

Upgrading from 1.3.0

If you previously installed dtf version 1.3.0, you can use the uninstall script to ensure there are no conflicts:

$ sudo ./uninstall_1_3.sh

Note that dtf version 1.3.1 does not require any changes to a user's $PATH, so you should remove any $PATH changes related to dtf.

Using dtf

Before using dtf, you'll need to enable USB debugging on your target device. If you're unsure of what this is, dtf is probably not the tool for you. Once it's enabled, update all Play applications, and connect the device to your PC. Assuming adb sees your device, we can create our project with the init built-in command:

$ mkdir MyProject
$ cd MyProject
$ dtf init

From here, you'll want to read up on each of the many modules that dtf supports. See the project Wiki for additional details on using dtf.

Licenses

dtf is licensed under the Apache License, Version 2.0, but contains additional code from other projects. Check the NOTICE file for additional projects and licensing.

Contributing & Building

If you're interested in building your own instance of dtf, you'll need a couple of dependences:

$ sudo apt-get install lintian python2.7 openjdk-8-jdk python-pip devscripts shellcheck
$ sudo pip install flake8 pylint pytest pytest-runner wheel

You can now build the project, which is currently limited to Debian '.deb' packages. To build dtf, run the following command from the project root:

$ ./gradlew clean useDebugApk makeDeb

Creating Content

The dtf man pages are a great place to start. The man pages for dtf-module(7), dtf-binary(7), dtf-library(7) and dtf-package(7) will provide additional insight on the structure of dtf content. If you're creating a module, it's a good idea to ensure that it passes all the checks with the dtf_check utility. More information can be found in the man pages for dtf-check(1). To check your new module:

$ dtf_check -sa my_cool_module

Questions & Comments

Please use the project's GitHub issue tracker for reporting bugs. For bugs related to a specific module, please use the issue tracker for that particular git repo.

dtf's People

Contributors

erikbarzdukas avatar jakev avatar jakev-mandiant avatar vladionescu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dtf's Issues

Support using adb over the wireless

I think by using adb over the wireless network you could save significant time pulling stuff from the device. I'll do some testing to confirm, but it should be easy to add.

Add "export" option to `pm`

Add export functionality to pm so that content can be shared.

$ dtf pm export

This will generate the XML for the "manifest.xml", and zip all content up.

tox integration

I'd like to add tox, and and use it to integrate with pytests/coverage.

"offline" device `init` fail

Sometimes, a device (or emulator) is offline when an installation is attempted, so a non-useful "$PATH not found" error occurs. I'd like to clean this up.

05:56:17 /DevTesting/test19$ dtf init
[Thu Feb 18 17:56:22 EST 2016] init/I - Project initialization started. 

Please connect the test device (press Enter to continue) 
[Thu Feb 18 17:56:23 EST 2016] init/I - Restarting adb... 
[Thu Feb 18 17:56:26 EST 2016] init/I - Waiting for a device to be connected... 
Got serial 'emulator-5554', is this correct? [Y/n] y
[Thu Feb 18 17:56:39 EST 2016] init/E - Unable to get $PATH variable! 
[Thu Feb 18 17:56:39 EST 2016] init/W - Exiting dtf initialization! 
05:56:39 /DevTesting/test19$ adb devices
List of devices attached
emulator-5554   offline

Launching bash module from non TOP directory fails

example:

04:15:55 /DevTesting/Fire/.drozer$ dtf usesperm
Traceback (most recent call last):
  File "/usr/local/bin/dtf", line 185, in <module>
    sys.exit(main())
  File "/usr/local/bin/dtf", line 175, in main
    rtn = pkg.launch_module(command_name, sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/dtf/packages.py", line 180, in launch_module
    return __launch_bash_module(module_path, args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/packages.py", line 112, in __launch_bash_module
    serial = prop.get_prop('Info', 'serial')
  File "/usr/local/lib/python2.7/dist-packages/dtf/properties.py", line 62, in get_prop
    raise PropertyError(err)
dtf.properties.PropertyError: Property section not found: Info

Handle 64 bit ARM on Lollipop

Newer devices will use ARM-64, which will store ODEX files in:

/system/framework/arm64/
/system/app/AppName/arm64/
/system/app-priv/AppName/arm64/

Bash modules that use relative paths will fail

Since dtf automatically changes to the TOP directory, it's not possible to use relative paths. I'm going to save the current directory before changing, so that modules that care can still use it.

Improve dtfClient <-> dtf Communications

Currently, the way to interact with the dtfClient application is not ideal. One of the main things it lacks is the ability to upload files directly to the dtfClient, which is important. This will help to solve that problem.

Steps:

  • Modify dtfClient to support UNIX socket for handling incoming connections
  • Modify dtf to be able to interact with the UNIX socket above

Add new DbLibrary class

Currently, a library within dtf is just an import-able Python module, however, there are now 5+ libraries that act as a database helper for storing content in the .dbs/ directory. As you can imagine, there is a ton of code reuse and it's really messy. The purpose of the DbLibrary class will be to:

  • Force libraries to follow a certain formatting
  • Remove all code in modules for checking if the DB exists, and setting up a remote project (think diffing)
  • Remove code that is reused in libraries (like having to implement a drop when you could use a drop_all())

Improve build process and allow upgrades

Currently, there is a single build, which is a .deb. This .deb includes the following:

  • man pages
  • The bash portions of dtf
  • The python portion of dtf
  • a bundle TAR containing tools (apktool, smali, etc.)

Some people may prefer to use their own apktool (which they already can), but ideally having a lite download is easier. When you download the lite version, there is a configuration that runs the first time you launch dtf.

What to do:

  • Standardize the build for downloading and building apktool, smali, etc
  • Move the included.tar from the python build to the .deb installer
  • Create a configuration wizard if dtf is not detected as built
  • Create .deb build flavors for full and lite [NOT ADDED]

UPDATE
I've decided against the full/lite option for the time being. I may investigate this later. Modifying the issue instead of closing as many of the pieces were implemented.

`client`'s download functionality returns unexpected response with small files

----------------------------- Captured stdout call -----------------------------
[Wed Jan 04 21:02:40 GMT 2017] client/I - Waiting for connected device... 
[Wed Jan 04 21:02:40 GMT 2017] client/I - Device connected! 
[Wed Jan 04 21:02:42 GMT 2017] dtfClient/D - Connecting to abstract socket... 
[Wed Jan 04 21:02:42 GMT 2017] dtfClient/D - Connected to abstract socket! 
[Wed Jan 04 21:02:42 GMT 2017] dtfClient/D - Sending filename to server 
[Wed Jan 04 21:02:42 GMT 2017] dtfClient/D - Filename sent. 
[Wed Jan 04 21:02:42 GMT 2017] dtfClient/D - File size from server: 25 
[Wed Jan 04 21:02:42 GMT 2017] dtfClient/D - Transfer complete! 
[Wed Jan 04 21:02:42 GMT 2017] client/E - Unknown response, cannot proceed.

Export fails when modules lack version

06:38:58 /DevTesting$ dtf pm export test.zip
Traceback (most recent call last):
  File "/usr/local/bin/dtf", line 185, in <module>
    sys.exit(main())
  File "/usr/local/bin/dtf", line 149, in main
    return pkg.launch_builtin_module('pm', sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/dtf/packages.py", line 145, in launch_builtin_module
    return __launch_python_module(launch_path, cmd, args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/packages.py", line 97, in __launch_python_module
    return mod_inst.run(args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/module.py", line 62, in run
    result = getattr(self, 'execute')(args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/core/cmds/pm.py", line 723, in execute
    rtn = self.do_export(args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/core/cmds/pm.py", line 235, in do_export
    rtn = self.generate_export_xml(export_items, export_manifest)
  File "/usr/local/lib/python2.7/dist-packages/dtf/core/cmds/pm.py", line 390, in generate_export_xml
    item_xml.attrib['majorVersion'] = item.major_version
  File "lxml.etree.pyx", line 2245, in lxml.etree._Attrib.__setitem__ (src/lxml/lxml.etree.c:58775)
  File "apihelpers.pxi", line 547, in lxml.etree._setAttributeValue (src/lxml/lxml.etree.c:19025)
  File "apihelpers.pxi", line 1393, in lxml.etree._utf8 (src/lxml/lxml.etree.c:26460)
TypeError: Argument must be bytes or unicode, got 'NoneType'

dtf drops .dtflog all over and expects to write to it

Due to changes in src/dtf/logging.py, dtf attempts to create a log file everytime the file is sourced. This fails if it unable to write to the file, such as the following example:

Traceback (most recent call last):
  File "/usr/local/bin/dtf", line 34, in <module>
    import dtf.packages as pkg
  File "/usr/local/lib/python2.7/dist-packages/dtf/packages.py", line 18, in <module>
    import dtf.core.packagemanager as pm
  File "/usr/local/lib/python2.7/dist-packages/dtf/core/packagemanager.py", line 19, in <module>
    import dtf.logging as log
  File "/usr/local/lib/python2.7/dist-packages/dtf/logging.py", line 38, in <module>
    LOG_FILE = open(LOG_FILE_NAME, 'a')
IOError: [Errno 13] Permission denied: '.dtflog'
bash: .: filename argument required
.: usage: . filename [arguments]

The only time a log file should be created is if dtf detects a project directory.

Various v1.2 --> v1.3 Broken things

Catch all for broken stuff.

  • $TOP no longer available to bash scripts (1829ef1)
  • Bash modules that are interactive (think dtf drozer) don't seem to be printing the output correctly. (da4f529)
  • $ANDROID_SERIAL is not set to bash scripts (1829ef1)
  • dtf_busybox doesn't know about $ANDROID_SERIAL. (1829ef1)
  • dtf status errors when executed (1829ef1)
  • TOP (part of pydtf/dtfcore.py) needs to be available to modules. (dacce92)
  • aapt doesn't seem to be working. 'aapt-23.0.1: error while loading shared libraries: libc++.so: cannot open shared object file: No such file or directory' (a5e4e4f)
  • In init, a new DtfAdb instance is not created once a serial number is established. This means that setting up a project with multiple devices connected will not work. (cb6b19a)
  • archive doesn't appear to work. (099f280)
  • stdout is no longer captured with "launch_module", which is a problem. (ff3408a)

why i type dtf -h ,can not work

root@ubuntu:/etc/ssh/dtf# ./setup.py
dtf installation started...
Completion output is already generated, skipping
dtf_core.sh output is already generated, skipping
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

root@ubuntu:/etc/bash_completion.d# dtf -h
No command 'dtf' found, did you mean:
Command 'dt' from package 'ditrack' (universe)
Command 'dtc' from package 'device-tree-compiler' (main)
Command 'tf' from package 'tf' (universe)
Command 'tf' from package 'tf5' (universe)
Command 'wtf' from package 'bsdgames' (universe)
Command 'dff' from package 'dff' (universe)
Command 'gtf' from package 'xserver-xorg-core' (main)

Explicitly state if a module is python/bash with installing

As I move towards supporting python 2.7+ and 3.x, I'll need to handle different envirnoments better. Certain modules will (at least initially) only work on 2.7, so dtf should refuse to work if it detects a 3.x instance. I'll modify the Item class and the tag to state what language the module is:

    <Item type="module"
          name="autosign"
          lang="shell"
          majorVersion="1"
          minorVersion="0"
          health="stable"
          author="Jake Valletta (jakev)"
          about="Sign an application with a test key."
          localName="modules/autosign" />

Default behavior will be python.

lang values can be:
-shell - just executed as a shell script
-python - will default to python3
-python2 - supports python 2 ONLY

`dtf status` ValueError

12:06:10 /DevTesting/test$ dtf status
Traceback (most recent call last):
  File "/usr/local/bin/dtf", line 185, in <module>
    sys.exit(main())
  File "/usr/local/bin/dtf", line 169, in main
    rtn = pkg.launch_builtin_module(command_name, sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/dtf/packages.py", line 124, in launch_builtin_module
    return __launch_python_module(launch_path, cmd, args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/packages.py", line 79, in __launch_python_module
    return mod_inst.run(args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/module.py", line 62, in run
    result = getattr(self, 'execute')(args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/core/cmds/status.py", line 35, in execute
    devices = self.adb.get_devices()
  File "/usr/local/lib/python2.7/dist-packages/dtf/adb.py", line 107, in get_devices
    serial, device_type = device.split('\t')
ValueError: need more than 1 value to unpack```

Coloring API support

I think it would be good to create a wrapper API for the python colored module (no support for bash). By letting modules use this, coloring can uniform and easy to customize.

Harden dtfClient

Since the dtfClient could pose as a security weakness on production devices, I'd like to add the following:

  • Sticky notification warning user that dtfClient is installed
  • Only allowing the shell user to use either of the services of the client

Non-existent import failure

07:54:43 /DevTesting$ echo "import garbage" > testing
08:24:20 /DevTesting$ dtf pm install --single module --name testing --auto
[Sat Nov 07 20:24:34 EST 2015] pm/I - Attempting to auto parse... 
Traceback (most recent call last):
  File "/usr/local/bin/dtf", line 185, in <module>
    sys.exit(main())
  File "/usr/local/bin/dtf", line 149, in main
    return pkg.launch_builtin_module('pm', sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/dtf/packages.py", line 124, in launch_builtin_module
    return __launch_python_module(launch_path, cmd, args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/packages.py", line 79, in __launch_python_module
    return mod_inst.run(args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/module.py", line 62, in run
    result = getattr(self, 'execute')(args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/core/cmds/pm.py", line 719, in execute
    rtn = self.do_install(args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/core/cmds/pm.py", line 136, in do_install
    item = self.auto_parse_module(parsed_args)
  File "/usr/local/lib/python2.7/dist-packages/dtf/core/cmds/pm.py", line 587, in auto_parse_module
    if packagemanager.is_python_module(local_name, install_name):
  File "/usr/local/lib/python2.7/dist-packages/dtf/core/packagemanager.py", line 336, in is_python_module
    imp.load_source(name, module_path)
  File "testing", line 1, in <module>
    import garbage
ImportError: No module named garbage

Remove `aapt` dependency

Since Ubuntu is the primary supported OS going forward, I'd like to remove needing to download the Android SDK. The only tools from it that dtf uses areadb and aapt, and adb is available from aptitude. It makes sense to include a copy of aapt as part of the "dtf-includes" structure. It just means that updating it is on me. This will affected several modules, but it is easy enough to implement.

Dtf fails to fallback to filesystem socket if abstract socket not available

Dtf creates an abstract socket with the adb command

adb forward localabstract:dtf_socket localabstract:dtf_socket

But if at the time adb is compiled __linux__ is not defined (can happen for various reasons) then adb will continue silently with the above command except it will create a filesystem socket instead, in the default path of /tmp.

This behavior is from https://android.googlesource.com/platform/system/core/+/master/libcutils/socket_local_client_unix.c#54

Since adb automatically has this behavior, dtf should follow it and attempt to connect to a filesystem socket if the abstract socket is not available. As a bonus, dtf will be more willing to work on OSes which do not support abstract sockets :).

For the record, the latest version of adb (1.0.32) shipped with android-tools-adb on Ubuntu 15 refuses to create abstract sockets and always falls back to a filesystem socket in /tmp. This is an issue with the Ubuntu 15 packaged adb. File info listed below for reference.

/usr/bin/adb: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=cda1373d7b9b9e4b57ab3068a12e31557b191b98, stripped

Going forward it would be a good idea to break dependency on the Ubuntu 15 packaged adb and either ship our own or pull from a reliable source.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.