/    Sign up×
Community /Pin to ProfileBookmark

What Do I Need To Look Into? (online interactive database build)

Hello everyone.
I’m new around here (so if I step out of line please feel free to alter as needed), and I read this was a good place to turn to for general developer community conversations.

_note: I chose “client-side” because that’s my largest hole, but I do have interest in the opinions regarding server-side in regards to databases (see below)_

I need help getting pointed in the right direction. I’m not looking for anyone to do anything for me.

As an amateur hobby, I collect astrosphere data that have been calculated by researchers and build that information into a catalog. This catalog, so far, has remained localized to myself and as such, remains rather limited and useless in scope. I would like to move this project online, sort of like the SIMBAD database, but with a bit of a difference in that it would also have a graphical plot chart which will show all astrosphere calculations (Y axis: astrosphere radii, X axis: star radii).

It would approximately appear something like this:
[upl-image-preview url=https://www.webdeveloper.com/assets/files/2019-08-09/1565377760-841784-image.png]

What this is, is a baseline (the black line of plots running at angle), and then calculations from various research (which tend to employ different models) plotted against it.

On top of this, I want to be able to build it so that the user interface allows the viewer to either view all data, all data with a set of parameters (e.g. stars of a given size, stars of given types, region, etc…), or to be able to compare different models (as different models tend to employ different equations and considerations in their estimates).
Additionally, while the data input can be manual (or manual upload), I want the “Profile” of a model to be automated. The “Profile” is a 5 point set of numbers which identify one model uniquely from any other model and contains the following values for astrosphere estimates: Minimum difference by factor of the model from the baseline, Maximum difference by factor of the model from the baseline, the Mean of the difference by factor of the model from the baseline, the p values of the T-Test and F-Test between the model and the baseline.

This will _also_ have a graphical representation if a user selects to drill into the profile view and compare different models.
It would approximately look something like this:
[upl-image-preview url=https://www.webdeveloper.com/assets/files/2019-08-09/1565378644-424723-image.png]

Lastly, I would like the user interface to be able to deliver tooltips on mouse-over of a data point and fill that tooltip with a limited set of quick information as selected by the user from an option menu (with a default set of options).

I am WELL aware that this will _not_ be a short project, nor will it be easy. Especially for myself, considering that I have limited time (as this is a hobby, and only one of two primary hobbies in my life), and don’t have a solid background in developing such a solution.
That said, I am _not_ averse to learning. I regularly self-teach (that’s what I did with astrophysics to get to where I needed to be to even begin this catalog), and my day-job requires a lot of self-teaching (data analyst is my day job).

I’m just looking for initial pointers as to _what_ I’m trying to look into for use in terms of both database solutions (this is mostly an opinion question, as I do know a variety of database solutions, but I use them for an entirely proprietary purpose through reporting software – not web-develop builds), and especially in terms of the user interface as that is where I’m definitely at the strongest disadvantage in knowledge.

I have some scratches of javascript knowledge, but it’s basically at the level of taking a snippet of code someone else offered up as a solution and modifying it a tad bit to fit.
I also have a decent working knowledge of HTML and CSS, as I’ve had need to employ those in my job on occasion for one-off solutions…though they are rather limited.
I’ve dabbled lightly in PHP a number of years back for a personal project, but to be honest, there’s cobwebs in that part of my brain for certain.
I do understand programming language syntax and structure as I have to use proprietary language, and a fair bit of executable scripting at my job.

I just don’t have any experience in building web-interfaces that are graphically, or even non-graphically, this advanced.

Any advice on what to look into would be very much appreciated! Thanks!

Cheers,
Jayson

to post a comment
Full-stack Developer

8 Comments(s)

Copy linkTweet thisAlerts:
@NogDogAug 09.2019 — My number one tip at this point is to break things down into individual modules/apps/whatever. Keep them "loosely coupled" so that you can work on them separately, and then "glue" them together as they are ready.

One specific suggestion that jumps out at me from the subject matter is to consider using Python for the server-side, data-management code, as Python has generally become the go-to language for data analysis in general and astrophysics data in particular -- not necessarily because other languages can't do it, but because there are so many relevant Python libraries out there you could use.
Copy linkTweet thisAlerts:
@JaysonAauthorAug 09.2019 — @NogDog#1607416 That's a great idea! On that note, I've been thinking about perhaps starting by just getting the database built sans-interface as step one.

And for step two, just getting a page to readout the data from a given row index selection.

Would you think this makes sense, given your advice - to basically just start firstly with building the server-side database solution, and then circle back around after that's put together?

On that note...one problem that I'm at a bit of an impasse on is whether to go the route of a structured or object database.

The former, to me at least, seems rather a bit easier to set up and get running, _but_ it has a rather large potential frustration at later stages of the build in not being as modular as the object database...I could have to restructure the database based on needs from the interface builds (or alternatively, find interface build limitations due to how I established the structure of the database).

The downside with the object database, for me, is I'm less familiar with it (and I already have quite a bit to do), and it seems a bit less organized in the long-term (the potential for disorder over the long-term seems high - I've witnessed some truly terrible object database organization practices where I work and the hell they've cost report analysts as a result), and (somewhat related the parenthetical) fewer external reporting tools seem quite up to speed with object databases (you _can_ plug-in, but my experience is that it's not usually plug-n-play; more plug-n-groan and eventually play), and I would like to build this in such a way that if someone _wants_ they could access the database via a connection to their preference of reporting engine/tool on their side if they have local tools they wish to use to employ for further processes than the catalog provides.

I know this is more of a non-client-side tangent of a question - sorry...this project really encompasses both.

Thanks for the feedback; I like the module idea! :)

Cheers,

Jayson
Copy linkTweet thisAlerts:
@NogDogAug 10.2019 — I'm mainly a server-side guy, and what database chops I have are mainly in the fairly traditional SQL/RDBMS line of things -- I've peeked a bit into graph databases as I see a potential for something that might be useful at work, but haven't really gotten into it yet. So, I can't claim to know what will be best for you, but a general rule I like to follow is to avoid premature optimization: don't decide you have to use the newest, most exotic technology out there with all sorts of configurations and modifications to make it as fast as possible, when it may be just fine with more traditional, mature, and comparatively simple approaches. :) Later, you can improve/optimize things as and when you determine there's a need. This gets back to keeping things as modular as possible: if you decides your PostgreSQL database is no longer the right approach and choose some NoSQL approach, graph DB, or whatever; you just* modify your database interface code to use the new approach, without having to touch the rest of the application.

_____________________

* Sure, "just" ;)
Copy linkTweet thisAlerts:
@JaysonAauthorAug 10.2019 — @NogDog#1607427 Thanks again. Sound advice.

Considering the target audience, as you noted, is a community largely steeped in Python and SQL databases, and because I'm more familiar with MySQL (I'm not proficient, but at least I've poked at one before - I don't set up databases at my job, I access them via connections to reporting tools), I'm thinking this is a good starting route: Python + MySQL.

Looking around, I found that pretty decent words are said about pythonanywhere as a fair place to get a feel for your project before taking bigger dives into other solutions. I looked them up and scanned them over and found that they have a decent array of guides for a variety of things.

One specifically caught my eye: https://blog.pythonanywhere.com/121/

It's a beginner level guide to making a Flask app for your Python using a MySQL database.

This sound like a decent starting point for what I'm looking to do.

Looking around, it _looks_ like Flask (with a little help) _can_ deliver charting functions when I get to that point down the road: https://blog.ruanbekker.com/blog/2017/12/14/graphing-pretty-charts-with-python-flask-and-chartjs/

So this _seems_ like the right path: Python + MySQL + Flask on pythonanywhere

I'd be interested in your thoughts on my tentative pick here (as well as anyone else reading along).

Thanks again!

Cheers,

Jayson
Copy linkTweet thisAlerts:
@NogDogAug 10.2019 — Sounds reasonable to me, just based on things I've heard -- not direct experience on my part. I know our Data Engineering team mainly uses Python these days, and I've heard them mention Flask. :) We use PostgreSQL for a few reasons (not all of them technical), but MySQL (or the offshoot MariaDB) will give you essentially the same functionality and performance as far as I know.
Copy linkTweet thisAlerts:
@JaysonAauthorAug 10.2019 — Nice!

It'll be a long road, but at least I finally have a rough roadmap and know what my toolbox looks like now.

Thanks a bunch for talking things out; it really helps to bounce things around! :)

Cheers,

Jayson
Copy linkTweet thisAlerts:
@alexstormAug 11.2019 — Jayson,

Hi. Getting that kind of results to auto appear on a website by user picking the parameters, is rather tricky. I actually have a similar project that also has to do with stars and creates graphs on the fly in the front end.

I use chart.js.org by Nick Downie. He is also very helpful.

Getting it to work is another matter. My project charts mission profile X= time and Y= speed. There is a second chart line that tracks time dilation. Speed an dilation take different values, it makes some sense to plot them together if you use light speed as 1 percent dilation as up 100% or 1. Anyway, I am digressing. It's a real bear to get this to work on a project.

The Mission Trip Calculator project with the chart is here: http://www.producerelease.com/sgc/#jcal

Audio is optional and fun, but also locks out another chart, until the music finishes.

Another graph I built uses prototype.js, raphael.js and ico.js: It's a configurator. Pick a few pull down options and then go to the bottom and update the graph. http://www.producerelease.com/passcode/passcodecalc.htm

Both handle everything in the front-end which means all the math and dbase is in the HTML files or .js. I will warn you that I think this type of complex graph work will have sort of limited appeal online. I hardly think I get more than a few visitors a year. It seems pretty logical that just stating your idea and putting a few Excel graph images will suit your needs of getting the project idea across. Plus I think back end data for graphs makes more sense. Making it front end only and interactive took a lot of effort. But, if you want to go that route, hopefully this helps.

The Trip Calculator chart can do over half a billion trip connections from the dbase and it does a manual mode, as well. Have a look at the illustrated guide on the Trip Calculator. It's amusingly, extremely complex. These project were more just to see if I could them, than trying to get a lot of people interested in using them.
Copy linkTweet thisAlerts:
@JaysonAauthorAug 12.2019 — Thank you @alexstorm ,

I will definitely be checking your site out (for web build reasons of course, but also I'm big into amateur science projects, so your work is really interesting to me!).

You make solid points.

To be clear, I'm actually looking at three approaches all coexisting:

1) Raw data available for download or (1.a) by db read-only connection from your choice of software.

2) Data search and readout like SIMBAD.

3) Front-end graphical interaction which keeps track of the data in comparison to the baseline and allows for easy casual comparisons without needing to hook into the raw data to just do a quick look.

The front-end will come last.

1 is first, then 2, then 1.a, then 3.

At least, that's what I've worked out as my plan so far.

Cheers,

Jayson
×

Success!

Help @JaysonA spread the word by sharing this article on Twitter...

Tweet This
Sign in
Forgot password?
Sign in with TwitchSign in with GithubCreate Account
about: ({
version: 0.1.9 BETA 3.29,
whats_new: community page,
up_next: more Davinci•003 tasks,
coming_soon: events calendar,
social: @webDeveloperHQ
});

legal: ({
terms: of use,
privacy: policy
});
changelog: (
version: 0.1.9,
notes: added community page

version: 0.1.8,
notes: added Davinci•003

version: 0.1.7,
notes: upvote answers to bounties

version: 0.1.6,
notes: article editor refresh
)...
recent_tips: (
tipper: @darkwebsites540,
tipped: article
amount: 10 SATS,

tipper: @Samric24,
tipped: article
amount: 1000 SATS,

tipper: Anonymous,
tipped: article
amount: 10 SATS,
)...