public
/
epg-grabber
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/master'

This commit is contained in:
Felix Jian 2023-08-03 22:55:40 +02:00
parent bf045dacae 3649f988f1
commit f74ebfe798
Signed by: flex
GPG Key ID: 2FB8FBECB390C227
16 changed files with 2966 additions and 2640 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
.github/workflows/test.yml vendored Normal file
View File

@ -0,0 +1,15 @@
name: test
on:
push:
branches:
- master
workflow_dispatch:
pull_request:
types: [opened, synchronize, reopened, edited]
jobs:
main:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- run: npm install
- run: npm test

8
.travis.yml
View File

@ -1,8 +0,0 @@
language: node_js
node_js:
- '10'
script:
- npm run lint
- npm run test

58
README.md
View File

@ -1,4 +1,4 @@
# EPG Grabber [![Build Status](https://app.travis-ci.com/freearhey/epg-grabber.svg?branch=master)](https://app.travis-ci.com/freearhey/epg-grabber) # EPG Grabber [![test](https://github.com/freearhey/epg-grabber/actions/workflows/test.yml/badge.svg)](https://github.com/freearhey/epg-grabber/actions/workflows/test.yml)
Node.js CLI tool for grabbing EPG from different websites. Node.js CLI tool for grabbing EPG from different websites.
@ -74,29 +74,31 @@ epg-grabber --config=example.com.config.js
Arguments: Arguments:
- `-c, --config`: path to config file - `-c, --config`: path to config file
- `-o, --output`: path to output file (default: 'guide.xml') - `-o, --output`: path to output file or path template (example: `guides/{site}.{lang}.xml`; default: `guide.xml`)
- `--channels`: path to list of channels (can be specified via config file) - `--channels`: path to list of channels; you can also use wildcard to specify the path to multiple files at once (example: `example.com_*.channels.xml`)
- `--lang`: set default language for all programs (default: 'en') - `--lang`: set default language for all programs (default: `en`)
- `--days`: number of days for which to grab the program (default: 1) - `--days`: number of days for which to grab the program (default: `1`)
- `--delay`: delay between requests in milliseconds (default: 3000) - `--delay`: delay between requests in milliseconds (default: `3000`)
- `--timeout`: set a timeout for each request in milliseconds (default: 5000) - `--timeout`: set a timeout for each request in milliseconds (default: `5000`)
- `--cache-ttl`: maximum time for storing each request in milliseconds (default: 0) - `--max-connections`: set a limit on the number of concurrent requests per site (default: `1`)
- `--gzip`: compress the output (default: false) - `--cache-ttl`: maximum time for storing each request in milliseconds (default: `0`)
- `--debug`: enable debug mode (default: false) - `--gzip`: compress the output (default: `false`)
- `--curl`: display current request as CURL (default: false) - `--debug`: enable debug mode (default: `false`)
- `--curl`: display current request as CURL (default: `false`)
- `--log`: path to log file (optional) - `--log`: path to log file (optional)
- `--log-level`: set the log level (default: 'info') - `--log-level`: set the log level (default: `info`)
## Site Config ## Site Config
```js ```js
module.exports = { module.exports = {
site: 'example.com', // site domain name (required) site: 'example.com', // site domain name (required)
output: 'example.com.guide.xml', // path to output file (default: 'guide.xml') output: 'example.com.guide.xml', // path to output file or path template (example: 'guides/{site}.{lang}.xml'; default: 'guide.xml')
channels: 'example.com.channels.xml', // path to channels.xml file (required) channels: 'example.com.channels.xml', // path to list of channels; you can also use an array to specify the path to multiple files at once (example: ['channels1.xml', 'channels2.xml']; required)
lang: 'fr', // default language for all programs (default: 'en') lang: 'fr', // default language for all programs (default: 'en')
days: 3, // number of days for which to grab the program (default: 1) days: 3, // number of days for which to grab the program (default: 1)
delay: 5000, // delay between requests (default: 3000) delay: 5000, // delay between requests (default: 3000)
maxConnections: 200, // limit on the number of concurrent requests (default: 1)
request: { // request options (details: https://github.com/axios/axios#request-config) request: { // request options (details: https://github.com/axios/axios#request-config)
@ -235,6 +237,34 @@ You can also specify the language and logo for each channel individually, like s
>France 24</channel> >France 24</channel>
``` ```
## How to use SOCKS proxy?
First, you need to install [socks-proxy-agent](https://www.npmjs.com/package/socks-proxy-agent):
```sh
npm install socks-proxy-agent
```
Then you can use it to create an agent that acts as a SOCKS proxy. Here is an example of how to do it with the Tor SOCKS proxy:
```js
const { SocksProxyAgent } = require('socks-proxy-agent')
const torProxyAgent = new SocksProxyAgent('socks://localhost:9050')
module.exports = {
site: 'example.com',
url: 'https://example.com/epg.json',
request: {
httpsAgent: torProxyAgent,
httpAgent: torProxyAgent
},
parser(context) {
// ...
}
}
```
## Contribution ## Contribution
If you find a bug or want to contribute to the code or documentation, you can help by submitting an [issue](https://github.com/freearhey/epg-grabber/issues) or a [pull request](https://github.com/freearhey/epg-grabber/pulls). If you find a bug or want to contribute to the code or documentation, you can help by submitting an [issue](https://github.com/freearhey/epg-grabber/issues) or a [pull request](https://github.com/freearhey/epg-grabber/pulls).

143
bin/epg-grabber.js
View File

@ -12,6 +12,7 @@ const { name, version, description } = require('../package.json')
const _ = require('lodash') const _ = require('lodash')
const dayjs = require('dayjs') const dayjs = require('dayjs')
const utc = require('dayjs/plugin/utc') const utc = require('dayjs/plugin/utc')
const { TaskQueue } = require('cwait')
dayjs.extend(utc) dayjs.extend(utc)
@ -21,11 +22,16 @@ program
.description(description) .description(description)
.requiredOption('-c, --config <config>', 'Path to [site].config.js file') .requiredOption('-c, --config <config>', 'Path to [site].config.js file')
.option('-o, --output <output>', 'Path to output file') .option('-o, --output <output>', 'Path to output file')
.option('--channels <channels>', 'Path to channels.xml file') .option('--channels <channels>', 'Path to list of channels')
.option('--lang <lang>', 'Set default language for all programs') .option('--lang <lang>', 'Set default language for all programs')
.option('--days <days>', 'Number of days for which to grab the program', parseNumber) .option('--days <days>', 'Number of days for which to grab the program', parseNumber)
.option('--delay <delay>', 'Delay between requests (in milliseconds)', parseNumber) .option('--delay <delay>', 'Delay between requests (in milliseconds)', parseNumber)
.option('--timeout <timeout>', 'Set a timeout for each request (in milliseconds)', parseNumber) .option('--timeout <timeout>', 'Set a timeout for each request (in milliseconds)', parseNumber)
.option(
'--max-connections <maxConnections>',
'Set a limit on the number of concurrent requests per site',
parseNumber
)
.option( .option(
'--cache-ttl <cacheTtl>', '--cache-ttl <cacheTtl>',
'Maximum time for storing each request (in milliseconds)', 'Maximum time for storing each request (in milliseconds)',
@ -53,6 +59,7 @@ async function main() {
curl: options.curl, curl: options.curl,
lang: options.lang, lang: options.lang,
delay: options.delay, delay: options.delay,
maxConnections: options.maxConnections,
request: { request: {
ignoreCookieErrors: true, ignoreCookieErrors: true,
} }
@ -60,62 +67,106 @@ async function main() {
if (options.timeout) config.request.timeout = options.timeout if (options.timeout) config.request.timeout = options.timeout
if (options.cacheTtl) config.request.cache.ttl = options.cacheTtl if (options.cacheTtl) config.request.cache.ttl = options.cacheTtl
if (options.channels) config.channels = options.channels
else if (config.channels)
config.channels = file.join(file.dirname(options.config), config.channels)
else throw new Error("The required 'channels' property is missing")
if (!config.channels) return logger.error('Path to [site].channels.xml is missing') if (options.channels) config.channels = options.channels
logger.info(`Loading '${config.channels}'...`)
let parsedChannels = []
if (config.channels) {
const dir = file.dirname(options.config)
let files = []
if (Array.isArray(config.channels)) {
files = config.channels.map(path => file.join(dir, path))
} else if (typeof config.channels === 'string') {
files = await file.list(config.channels)
} else {
throw new Error('The "channels" attribute must be of type array or string')
}
for (let filepath of files) {
logger.info(`Loading '${filepath}'...`)
const channelsXML = file.read(filepath)
const { channels } = parseChannels(channelsXML)
parsedChannels = parsedChannels.concat(channels)
}
} else throw new Error('Path to "channels" is missing')
const grabber = new EPGGrabber(config) const grabber = new EPGGrabber(config)
const channelsXML = file.read(config.channels) let template = options.output || config.output
const { channels } = parseChannels(channelsXML) const variables = file.templateVariables(template)
let programs = [] const groups = _.groupBy(parsedChannels, channel => {
let i = 1 let groupId = ''
let days = config.days || 1 for (let key in channel) {
const total = channels.length * days if (variables.includes(key)) {
const utcDate = getUTCDate() groupId += channel[key]
const dates = Array.from({ length: days }, (_, i) => utcDate.add(i, 'd')) }
for (let channel of channels) {
if (!channel.logo && config.logo) {
channel.logo = await grabber.loadLogo(channel)
} }
for (let date of dates) { return groupId
await grabber })
.grab(channel, date, (data, err) => {
logger.info(
`[${i}/${total}] ${config.site} - ${data.channel.id} - ${dayjs
.utc(data.date)
.format('MMM D, YYYY')} (${data.programs.length} programs)`
)
if (err) logger.error(err.message) for (let groupId in groups) {
const channels = groups[groupId]
let programs = []
let i = 1
let days = config.days || 1
const maxConnections = config.maxConnections || 1
const total = channels.length * days
const utcDate = getUTCDate()
const dates = Array.from({ length: days }, (_, i) => utcDate.add(i, 'd'))
const taskQueue = new TaskQueue(Promise, maxConnections)
if (i < total) i++ let queue = []
}) for (let channel of channels) {
.then(results => { if (!channel.logo && config.logo) {
programs = programs.concat(results) channel.logo = await grabber.loadLogo(channel)
}) }
for (let date of dates) {
queue.push({ channel, date })
}
} }
await Promise.all(
queue.map(
taskQueue.wrap(async ({ channel, date }) => {
await grabber
.grab(channel, date, (data, err) => {
logger.info(
`[${i}/${total}] ${config.site} - ${data.channel.xmltv_id} - ${dayjs
.utc(data.date)
.format('MMM D, YYYY')} (${data.programs.length} programs)`
)
if (err) logger.error(err.message)
if (i < total) i++
})
.then(results => {
programs = programs.concat(results)
})
})
)
)
programs = _.uniqBy(programs, p => p.start + p.channel)
const xml = generateXMLTV({ channels, programs })
let outputPath = file.templateFormat(template, channels[0])
if (options.gzip) {
outputPath = outputPath || 'guide.xml.gz'
const compressed = await gzip(xml)
file.write(outputPath, compressed)
} else {
outputPath = outputPath || 'guide.xml'
file.write(outputPath, xml)
}
logger.info(`File '${outputPath}' successfully saved`)
} }
programs = _.uniqBy(programs, p => p.start + p.channel)
const xml = generateXMLTV({ channels, programs })
let outputPath = options.output || config.output
if (options.gzip) {
outputPath = outputPath || 'guide.xml.gz'
const compressed = await gzip(xml)
file.write(outputPath, compressed)
} else {
outputPath = outputPath || 'guide.xml'
file.write(outputPath, xml)
}
logger.info(`File '${outputPath}' successfully saved`)
logger.info('Finish') logger.info('Finish')
} }

4799
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

4
package.json
View File

@ -1,6 +1,6 @@
{ {
"name": "epg-grabber", "name": "epg-grabber",
"version": "0.29.8", "version": "0.32.0",
"description": "Node.js CLI tool for grabbing EPG from different sites", "description": "Node.js CLI tool for grabbing EPG from different sites",
"main": "src/index.js", "main": "src/index.js",
"preferGlobal": true, "preferGlobal": true,
@ -34,8 +34,10 @@
"axios-mock-adapter": "^1.20.0", "axios-mock-adapter": "^1.20.0",
"commander": "^7.1.0", "commander": "^7.1.0",
"curl-generator": "^0.2.0", "curl-generator": "^0.2.0",
"cwait": "^1.1.2",
"dayjs": "^1.10.4", "dayjs": "^1.10.4",
"epg-parser": "^0.1.6", "epg-parser": "^0.1.6",
"fs-extra": "^11.1.1",
"glob": "^7.1.6", "glob": "^7.1.6",
"lodash": "^4.17.21", "lodash": "^4.17.21",
"node-gzip": "^1.1.2", "node-gzip": "^1.1.2",

30
src/Channel.js
View File

@ -1,23 +1,23 @@
class Channel { class Channel {
constructor(c) { constructor(c) {
const data = { const data = {
id: c.id || c.xmltv_id, xmltv_id: c.xmltv_id,
name: c.name, name: c.name,
site: c.site || '', site: c.site || '',
site_id: c.site_id, site_id: c.site_id,
lang: c.lang || '', lang: c.lang || '',
logo: c.logo || '', logo: c.logo || '',
url: c.url || toURL(c.site) url: c.url || toURL(c.site)
} }
for (let key in data) { for (let key in data) {
this[key] = data[key] this[key] = data[key]
} }
} }
} }
module.exports = Channel module.exports = Channel
function toURL(site) { function toURL(site) {
return site ? `https://${site}` : '' return site ? `https://${site}` : ''
} }

190
src/Program.js
View File

@ -3,140 +3,140 @@ const { toArray, toUnix, parseNumber } = require('./utils')
const Channel = require('./Channel') const Channel = require('./Channel')
class Program { class Program {
constructor(p, c) { constructor(p, c) {
if (!(c instanceof Channel)) { if (!(c instanceof Channel)) {
throw new Error('The second argument in the constructor must be the "Channel" class') throw new Error('The second argument in the constructor must be the "Channel" class')
} }
const data = { const data = {
site: c.site || '', site: c.site || '',
channel: c.id || '', channel: c.xmltv_id || '',
titles: toArray(p.titles || p.title).map(text => toTextObject(text, c.lang)), titles: toArray(p.titles || p.title).map(text => toTextObject(text, c.lang)),
sub_titles: toArray(p.sub_titles || p.sub_title).map(text => toTextObject(text, c.lang)), sub_titles: toArray(p.sub_titles || p.sub_title).map(text => toTextObject(text, c.lang)),
descriptions: toArray(p.descriptions || p.description || p.desc).map(text => descriptions: toArray(p.descriptions || p.description || p.desc).map(text =>
toTextObject(text, c.lang) toTextObject(text, c.lang)
), ),
icon: toIconObject(p.icon), icon: toIconObject(p.icon),
episodeNumbers: p.episodeNum || p.episodeNumbers || getEpisodeNumbers(p.season, p.episode), episodeNumbers: p.episodeNum || p.episodeNumbers || getEpisodeNumbers(p.season, p.episode),
date: p.date ? toUnix(p.date) : null, date: p.date ? toUnix(p.date) : null,
start: p.start ? toUnix(p.start) : null, start: p.start ? toUnix(p.start) : null,
stop: p.stop ? toUnix(p.stop) : null, stop: p.stop ? toUnix(p.stop) : null,
urls: toArray(p.urls || p.url).map(toUrlObject), urls: toArray(p.urls || p.url).map(toUrlObject),
ratings: toArray(p.ratings || p.rating).map(toRatingObject), ratings: toArray(p.ratings || p.rating).map(toRatingObject),
categories: toArray(p.categories || p.category).map(text => toTextObject(text, c.lang)), categories: toArray(p.categories || p.category).map(text => toTextObject(text, c.lang)),
directors: toArray(p.directors || p.director).map(toPersonObject), directors: toArray(p.directors || p.director).map(toPersonObject),
actors: toArray(p.actors || p.actor).map(toPersonObject), actors: toArray(p.actors || p.actor).map(toPersonObject),
writers: toArray(p.writers || p.writer).map(toPersonObject), writers: toArray(p.writers || p.writer).map(toPersonObject),
adapters: toArray(p.adapters || p.adapter).map(toPersonObject), adapters: toArray(p.adapters || p.adapter).map(toPersonObject),
producers: toArray(p.producers || p.producer).map(toPersonObject), producers: toArray(p.producers || p.producer).map(toPersonObject),
composers: toArray(p.composers || p.composer).map(toPersonObject), composers: toArray(p.composers || p.composer).map(toPersonObject),
editors: toArray(p.editors || p.editor).map(toPersonObject), editors: toArray(p.editors || p.editor).map(toPersonObject),
presenters: toArray(p.presenters || p.presenter).map(toPersonObject), presenters: toArray(p.presenters || p.presenter).map(toPersonObject),
commentators: toArray(p.commentators || p.commentator).map(toPersonObject), commentators: toArray(p.commentators || p.commentator).map(toPersonObject),
guests: toArray(p.guests || p.guest).map(toPersonObject) guests: toArray(p.guests || p.guest).map(toPersonObject)
} }
for (let key in data) { for (let key in data) {
this[key] = data[key] this[key] = data[key]
} }
} }
} }
module.exports = Program module.exports = Program
function toTextObject(text, lang) { function toTextObject(text, lang) {
if (typeof text === 'string') { if (typeof text === 'string') {
return { value: text, lang } return { value: text, lang }
} }
return { return {
value: text.value, value: text.value,
lang: text.lang lang: text.lang
} }
} }
function toPersonObject(person) { function toPersonObject(person) {
if (typeof person === 'string') { if (typeof person === 'string') {
return { return {
value: person, value: person,
url: [], url: [],
image: [] image: []
} }
} }
return { return {
value: person.value, value: person.value,
url: toArray(person.url).map(toUrlObject), url: toArray(person.url).map(toUrlObject),
image: toArray(person.image).map(toImageObject) image: toArray(person.image).map(toImageObject)
} }
} }
function toImageObject(image) { function toImageObject(image) {
if (typeof image === 'string') return { type: '', size: '', orient: '', system: '', value: image } if (typeof image === 'string') return { type: '', size: '', orient: '', system: '', value: image }
return { return {
type: image.type || '', type: image.type || '',
size: image.size || '', size: image.size || '',
orient: image.orient || '', orient: image.orient || '',
system: image.system || '', system: image.system || '',
value: image.value value: image.value
} }
} }
function toRatingObject(rating) { function toRatingObject(rating) {
if (typeof rating === 'string') return { system: '', icon: '', value: rating } if (typeof rating === 'string') return { system: '', icon: '', value: rating }
return { return {
system: rating.system || '', system: rating.system || '',
icon: rating.icon || '', icon: rating.icon || '',
value: rating.value || '' value: rating.value || ''
} }
} }
function toUrlObject(url) { function toUrlObject(url) {
if (typeof url === 'string') return { system: '', value: url } if (typeof url === 'string') return { system: '', value: url }
return { return {
system: url.system || '', system: url.system || '',
value: url.value || '' value: url.value || ''
} }
} }
function toIconObject(icon) { function toIconObject(icon) {
if (!icon) return { src: '' } if (!icon) return { src: '' }
if (typeof icon === 'string') return { src: icon } if (typeof icon === 'string') return { src: icon }
return { return {
src: icon.src || '' src: icon.src || ''
} }
} }
function getEpisodeNumbers(s, e) { function getEpisodeNumbers(s, e) {
s = parseNumber(s) s = parseNumber(s)
e = parseNumber(e) e = parseNumber(e)
return [createXMLTVNS(s, e), createOnScreen(s, e)].filter(Boolean) return [createXMLTVNS(s, e), createOnScreen(s, e)].filter(Boolean)
} }
function createXMLTVNS(s, e) { function createXMLTVNS(s, e) {
if (!e) return null if (!e) return null
s = s || 1 s = s || 1
return { return {
system: 'xmltv_ns', system: 'xmltv_ns',
value: `${s - 1}.${e - 1}.0/1` value: `${s - 1}.${e - 1}.0/1`
} }
} }
function createOnScreen(s, e) { function createOnScreen(s, e) {
if (!e) return null if (!e) return null
s = s || 1 s = s || 1
s = padStart(s, 2, '0') s = padStart(s, 2, '0')
e = padStart(e, 2, '0') e = padStart(e, 2, '0')
return { return {
system: 'onscreen', system: 'onscreen',
value: `S${s}E${e}` value: `S${s}E${e}`
} }
} }

198
src/client.js
View File

@ -13,125 +13,125 @@ module.exports.parseResponse = parseResponse
let timeout let timeout
function create(config) { function create(config) {
const client = setupCache( const client = setupCache(
axios.create({ axios.create({
headers: { headers: {
'User-Agent': 'User-Agent':
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36 Edg/79.0.309.71' 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36 Edg/79.0.309.71'
} }
}) })
) )
client.interceptors.request.use( client.interceptors.request.use(
function (request) { function (request) {
if (config.debug) { if (config.debug) {
console.log('Request:', JSON.stringify(request, null, 2)) console.log('Request:', JSON.stringify(request, null, 2))
} }
return request return request
}, },
function (error) { function (error) {
return Promise.reject(error) return Promise.reject(error)
} }
) )
client.interceptors.response.use( client.interceptors.response.use(
function (response) { function (response) {
if (config.debug) { if (config.debug) {
const data = const data =
isObject(response.data) || Array.isArray(response.data) isObject(response.data) || Array.isArray(response.data)
? JSON.stringify(response.data) ? JSON.stringify(response.data)
: response.data.toString() : response.data.toString()
console.log( console.log(
'Response:', 'Response:',
JSON.stringify( JSON.stringify(
{ {
headers: response.headers, headers: response.headers,
data, data,
cached: response.cached cached: response.cached
}, },
null, null,
2 2
) )
) )
} }
clearTimeout(timeout) clearTimeout(timeout)
return response return response
}, },
function (error) { function (error) {
clearTimeout(timeout) clearTimeout(timeout)
return Promise.reject(error) return Promise.reject(error)
} }
) )
return client return client
} }
async function buildRequest({ channel, date, config }) { async function buildRequest({ channel, date, config }) {
const CancelToken = axios.CancelToken const CancelToken = axios.CancelToken
const source = CancelToken.source() const source = CancelToken.source()
const request = { ...config.request } const request = { ...config.request }
timeout = setTimeout(() => { timeout = setTimeout(() => {
source.cancel('Connection timeout') source.cancel('Connection timeout')
}, request.timeout) }, request.timeout)
request.headers = await getRequestHeaders({ channel, date, config }) request.headers = await getRequestHeaders({ channel, date, config })
request.url = await getRequestUrl({ channel, date, config }) request.url = await getRequestUrl({ channel, date, config })
request.data = await getRequestData({ channel, date, config }) request.data = await getRequestData({ channel, date, config })
request.cancelToken = source.token request.cancelToken = source.token
if (config.curl) { if (config.curl) {
const curl = CurlGenerator({ const curl = CurlGenerator({
url: request.url, url: request.url,
method: request.method, method: request.method,
headers: request.headers, headers: request.headers,
body: request.data body: request.data
}) })
console.log(curl) console.log(curl)
} }
return request return request
} }
function parseResponse(response) { function parseResponse(response) {
return { return {
content: response.data.toString(), content: response.data.toString(),
buffer: response.data, buffer: response.data,
headers: response.headers, headers: response.headers,
request: response.request, request: response.request,
cached: response.cached cached: response.cached
} }
} }
async function getRequestHeaders({ channel, date, config }) { async function getRequestHeaders({ channel, date, config }) {
if (typeof config.request.headers === 'function') { if (typeof config.request.headers === 'function') {
const headers = config.request.headers({ channel, date }) const headers = config.request.headers({ channel, date })
if (isPromise(headers)) { if (isPromise(headers)) {
return await headers return await headers
} }
return headers return headers
} }
return config.request.headers || null return config.request.headers || null
} }
async function getRequestData({ channel, date, config }) { async function getRequestData({ channel, date, config }) {
if (typeof config.request.data === 'function') { if (typeof config.request.data === 'function') {
const data = config.request.data({ channel, date }) const data = config.request.data({ channel, date })
if (isPromise(data)) { if (isPromise(data)) {
return await data return await data
} }
return data return data
} }
return config.request.data || null return config.request.data || null
} }
async function getRequestUrl({ channel, date, config }) { async function getRequestUrl({ channel, date, config }) {
if (typeof config.url === 'function') { if (typeof config.url === 'function') {
const url = config.url({ channel, date }) const url = config.url({ channel, date })
if (isPromise(url)) { if (isPromise(url)) {
return await url return await url
} }
return url return url
} }
return config.url return config.url
} }

47
src/file.js
View File

@ -1,33 +1,62 @@
const fs = require('fs') const fs = require('fs')
const path = require('path') const path = require('path')
const glob = require('glob')
module.exports.list = list
module.exports.read = read module.exports.read = read
module.exports.write = write module.exports.write = write
module.exports.resolve = resolve module.exports.resolve = resolve
module.exports.join = join module.exports.join = join
module.exports.dirname = dirname module.exports.dirname = dirname
module.exports.templateVariables = templateVariables
module.exports.templateFormat = templateFormat
function list(pattern) {
return new Promise(resolve => {
glob(pattern, function (err, files) {
resolve(files)
})
})
}
function read(filepath) { function read(filepath) {
return fs.readFileSync(path.resolve(filepath), { encoding: 'utf-8' }) return fs.readFileSync(path.resolve(filepath), { encoding: 'utf-8' })
} }
function write(filepath, data) { function write(filepath, data) {
const dir = path.resolve(path.dirname(filepath)) const dir = path.resolve(path.dirname(filepath))
if (!fs.existsSync(dir)) { if (!fs.existsSync(dir)) {
fs.mkdirSync(dir, { recursive: true }) fs.mkdirSync(dir, { recursive: true })
} }
fs.writeFileSync(path.resolve(filepath), data) fs.writeFileSync(path.resolve(filepath), data)
} }
function resolve(filepath) { function resolve(filepath) {
return path.resolve(filepath) return path.resolve(filepath)
} }
function join(path1, path2) { function join(path1, path2) {
return path.join(path1, path2) return path.join(path1, path2)
} }
function dirname(filepath) { function dirname(filepath) {
return path.dirname(filepath) return path.dirname(filepath)
}
function templateVariables(template) {
const match = template.match(/{[^}]+}/g)
return Array.isArray(match) ? match.map(s => s.substring(1, s.length - 1)) : []
}
function templateFormat(template, obj) {
let output = template
for (let key in obj) {
const regex = new RegExp(`{${key}}`, 'g')
const value = obj[key] || undefined
output = output.replace(regex, value)
}
return output
} }

2
src/xmltv.js
View File

@ -29,7 +29,7 @@ function createElements(channels, programs, date) {
...channels.map(channel => { ...channels.map(channel => {
return ( return (
'\r\n' + '\r\n' +
el('channel', { id: channel.id }, [ el('channel', { id: channel.xmltv_id }, [
el('display-name', {}, [escapeString(channel.name)]), el('display-name', {}, [escapeString(channel.name)]),
el('icon', { src: channel.logo }), el('icon', { src: channel.logo }),
el('url', {}, [channel.url]) el('url', {}, [channel.url])

23
tests/Channel.test.js
View File

@ -12,28 +12,7 @@ it('can create new Channel', () => {
expect(channel).toMatchObject({ expect(channel).toMatchObject({
name: '1 TV', name: '1 TV',
id: '1TV.com', xmltv_id: '1TV.com',
site_id: '1',
site: 'example.com',
url: 'https://example.com',
lang: 'fr',
logo: 'https://example.com/logos/1TV.png'
})
})
it('can create channel from exist object', () => {
const channel = new Channel({
name: '1 TV',
id: '1TV.com',
site_id: '1',
site: 'example.com',
lang: 'fr',
logo: 'https://example.com/logos/1TV.png'
})
expect(channel).toMatchObject({
name: '1 TV',
id: '1TV.com',
site_id: '1', site_id: '1',
site: 'example.com', site: 'example.com',
url: 'https://example.com', url: 'https://example.com',

2
tests/__data__/input/example.config.js
View File

@ -7,7 +7,7 @@ module.exports = {
site: 'example.com', site: 'example.com',
days: 2, days: 2,
channels: 'example.channels.xml', channels: 'example.channels.xml',
output: 'tests/output/guide.xml', output: 'tests/__data__/output/guide.xml',
url: () => 'http://example.com/20210319/1tv.json', url: () => 'http://example.com/20210319/1tv.json',
request: { request: {
method: 'POST', method: 'POST',

7
tests/__data__/input/example_2.channels.xml Normal file
View File

@ -0,0 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<site site="example2.com">
<channels>
<channel xmltv_id="3TV.com" site_id="3">3 TV</channel>
<channel xmltv_id="4TV.com" site_id="4">4 TV</channel>
</channels>
</site>

32
tests/__data__/input/example_channels.config.js Normal file
View File

@ -0,0 +1,32 @@
const dayjs = require('dayjs')
const utc = require('dayjs/plugin/utc')
dayjs.extend(utc)
module.exports = {
site: 'example.com',
days: 2,
channels: ['example.channels.xml', 'example_2.channels.xml'],
output: 'tests/__data__/output/guide.xml',
url: () => 'http://example.com/20210319/1tv.json',
request: {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Cookie: 'abc=123'
},
data() {
return { accountID: '123' }
}
},
parser: () => {
return [
{
title: 'Program1',
start: 1640995200000,
stop: 1640998800000
}
]
},
logo: () => 'http://example.com/logos/1TV.png?x=шеллы&sid=777'
}

48
tests/bin.test.js
View File

@ -1,5 +1,5 @@
const { execSync } = require('child_process') const { execSync } = require('child_process')
const fs = require('fs') const fs = require('fs-extra')
const path = require('path') const path = require('path')
const epgParser = require('epg-parser') const epgParser = require('epg-parser')
@ -11,6 +11,10 @@ function stdoutResultTester(stdout) {
}) })
} }
beforeEach(() => {
fs.emptyDirSync('tests/__data__/output')
})
it('can load config', () => { it('can load config', () => {
const stdout = execSync( const stdout = execSync(
`node ${pwd}/bin/epg-grabber.js --config=tests/__data__/input/example.config.js --delay=0`, `node ${pwd}/bin/epg-grabber.js --config=tests/__data__/input/example.config.js --delay=0`,
@ -62,6 +66,26 @@ it('can generate gzip version', () => {
) )
}) })
it('can produce multiple outputs', () => {
const stdout = execSync(
`node ${pwd}/bin/epg-grabber.js \
--config=tests/__data__/input/mini.config.js \
--channels=tests/__data__/input/example.channels.xml \
--output=tests/__data__/output/{lang}/{xmltv_id}.xml`,
{
encoding: 'utf8'
}
)
expect(stdoutResultTester(stdout)).toBe(true)
expect(stdout.includes("File 'tests/__data__/output/fr/1TV.com.xml' successfully saved")).toBe(
true
)
expect(
stdout.includes("File 'tests/__data__/output/undefined/2TV.com.xml' successfully saved")
).toBe(true)
})
it('removes duplicates of the program', () => { it('removes duplicates of the program', () => {
const stdout = execSync( const stdout = execSync(
`node ${pwd}/bin/epg-grabber.js \ `node ${pwd}/bin/epg-grabber.js \
@ -81,3 +105,25 @@ it('removes duplicates of the program', () => {
expect(output.programs).toEqual(expected.programs) expect(output.programs).toEqual(expected.programs)
}) })
it('can load multiple "channels.xml" files at once', () => {
const stdout = execSync(
`node ${pwd}/bin/epg-grabber.js --config=tests/__data__/input/example.config.js --channels=tests/__data__/input/example*.channels.xml --timeout=1`,
{
encoding: 'utf8'
}
)
expect(stdoutResultTester(stdout)).toBe(true)
})
it('can parse list of "channels.xml" from array', () => {
const stdout = execSync(
`node ${pwd}/bin/epg-grabber.js --config=tests/__data__/input/example_channels.config.js --timeout=1`,
{
encoding: 'utf8'
}
)
expect(stdoutResultTester(stdout)).toBe(true)
})