radiot_dumper's Introduction

Radio-T Dumper

A.K.A. Http copier, because you can use it with different sites, and it will successfully detect file type and download content.

In an endless loop with a break on delay (to avoid creating a heavy load on the target server), the application will wait for a OK 200 response, after which it will copy the response body into the generated file.

Features

File type detection (without losing bytes)
Waiting for broadcast availability
Delay for avoiding creating a heavy load on the target server
Automatic generation of human-readable file names
Unlimited number of files (Limited by disk space)
Unit tests

Configuration

You can use dumper.yml file or environment variables for configuration.

yml	env	default	description
source_url	SOURCE_URL		Address from where to download information
file_prefix	FILE_PREFIX		Prefix for files
schedule	SCHEDULE		Schedule for downloading information in cron format
duration	DURATION		Duration of the dump
delay	DELAY	5s	Delay between request in case of interruption of the stream
file_date_format	FILE_DATE_FORMAT	02_01_2006	Date format for files
output_directory	OUTPUT_DIRECTORY	.	Directory where to save files
log_level	LOG_LEVEL	info	Log level

Run with docker compose

Before starting the application, you can change output directory volume in docker-compose.yml file from ./records to your desired directory. In container all files will be in the /tmp/output directory.

To start the application, run:

docker-compose up -d

radiot_dumper's People

Contributors

Stargazers

Watchers

radiot_dumper's Issues

StreamCopier does not close resp.Body when done

I just noticed resp.Body.Close() is never called by StreamCopier here.

https://cs.opensource.google/go/go/+/refs/tags/go1.18.4:src/net/http/response.go;l=61-64
https://pkg.go.dev/net/http#example-Get

Network interruptions

Add tests that simulate network interruptions and code, that handle network problems

StreamCopier.CopyStream does not copy the header of the file into the output

After the discussion we had in gabriel-vasile/mimetype#310, I had a look at the copier.StreamCopier and I think there is a problem.

step1: extensionFromBody will read file header in order to detect extension. It reads from a bufio.Reader which, in turn, reads from resp.Body.
step2: Later, io.Copy(output, resp.Body) is called and, because the header was already read in step1, it is skipped.

You can confirm the problem by saving a URL twice:

once using radiot_dumper (make sure extensionFromBody is called when you test)
second time use the browser to save the file.
Then, compare the file saved with radiot_dumper and the file saved using the browser. The radiot_dumper file will be smaller because it is missing the header.

Recommend Projects