Recently I had an idea for a research tool, I realized I'd need to be able to find websites programmatically and the webs data is behind the largest corporations, not something we reliably have access to for being able to develop around.

So it got me thinking, how lightweight could you first build a crawler if you only cared about getting meta data (titles, description and the url)? Then it got me thinking, could we not share the load, create a distributed peer to peer network and make this data decentralized?

So what I have built is the open web directory. A network that crawls and builds a yellow pages like directory for web pages online.

You can setup your own node and contribute to the network by simply running:

git clone https://github.com/idev-games/the-open-web-directory.git cd the-open-web-directory npm start 

And open port 80 for that device by port forwarding.

I think this is going to be a fascinating experiment, it's early days so expect bugs and issues but give it ago and let's see if we can index the web.

Check out more info here:

https://github.com/iDev-Games/The-Open-Web-Directory

The site listed on github is the live front end (you can host the html files in the public folder anywhere to open another frontend on the web).

Hello, I have been thinking about a project I‘d like to try out, where I have like a Database, an api server and a web server. It‘s not too large, and mostly to learn about how they all interact. The issue is, I would prefer not having to pay for multiple servers to host everything. Is there a way to simulate how it would be in real life on a single machine?

Hey guys, just after opinions really. If you’re launching a somewhat large site, with quite a bit of stuff going on, would you rely on your own testing or would you suggest getting a third party tester? It’s mostly for things like UX I’m wanting to test. I know have I would expect things to work but being the sole developer I think I may have gone a bit blind to somethings. I have friends do bits of pieces in terms of checking things but I wonder if the consensus would be to just get an actual tester to go through everything, testing the functionality all works, different variations, ensure the UX is as good as it can be etc or do you think I’m overthinking it?

so im working on my final year project report (pfe) (projet fin d'annee = Final year project which is literlly a big project to final my 6 years, in uni ) and i had to do the whole uml thing, use case diagram, class diagram, sequence diagrams for the main flows

and while i was doing it i kept asking myself, how do i actually know if what im drawing is "correct"

like the global logic of the app is obviously fixed, i know what the app does and how it should work, but when it comes to modeling it visually there are like ten different ways to represent the same thing. someone else looking at the same requirements could come up with a completely different class breakdown or a different way of grouping use cases, and technically neither version would be "wrong"

so when the jury reviews it, are they checking for one specific correct answer, or are they more just checking if your model is internally consistent and reflects a reasonable understanding of the system

anyone who's been through something similar, academic reviews, code reviews, whatever, how do you even validate a diagram when there's no single ground truth to compare it against

side note for context since some people outside the french education system might not know this, the PFE report is usually structured around sprints, like sprint 0 for environment setup, then sprint 1, 2, 3 etc each with its own objective and backlog, and for every single sprint you're expected to redo a use case diagram, a class diagram and sometimes sequence diagrams specific to that sprint

i pulled up an example report just to check and it had 6 sprints and something like 57 figures total, most of them being use case, class or sequence diagrams repeated per sprint. so its not like you draw one diagram for the whole app, you basically draw a mini version of the same three diagrams every sprint, which is part of why i started wondering how strict the "correctness" of each one actually needs to be

Does anyone know how to add completely secure video calls to a website? I’m struggling to find the best way.

Additionally, how do I then host this for free / as low cost as possible.

If anyone has advice that would be great!