Repository for http://griesheim-transparent.de - A search engine for documents of the Griesheim (Germany, Hessen, Darmstadt-Dieburg) city parliament.
- scraper: Scrapy-based webscraper for the "Ratsinformationssystem" aka. sessionnet https://sessionnet.owl-it.de/griesheim/bi/info.asp
- frontend: Django Frontend and management jobs for analysis
- solr: Solr Search Platform configuration
The full service is build on several microservices required at indexing time and for data storage (see docker compose file for details):
- A postgresql database to store scraped metadata.
- Tika for pdf metadata extraction, text extraction and OCR.
- fpurchess/preview-service for thumbnail generation.
- gotenberg to convert different file formats to pdf.
- The scraper docker image runs a cron job to scrape the sessionnet regulary and stores metadata+binary files stored to postgresql and the datastore.
- The frontend management task is also run in a cron job to periodically sync the scraped data into the solr index for searching. This includes:
- Combining metadata from scraped meetings, meeting agendas, consultations etc.
- Converting non-pdfs to pdf
- Extracting document metadata and content from pdfs with pdfact, tika and/or tika+tesseract (ocr)
- Generating preview images with the preview-service
- The frontend django app makes the data available to the user by queyring the solr search platform
MIT